You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@druid.apache.org by "kfaraz (via GitHub)" <gi...@apache.org> on 2025/07/21 11:23:31 UTC

[PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

kfaraz opened a new pull request, #18302:
URL: https://github.com/apache/druid/pull/18302

   ### Description
   
   This contains the same changes as #18265.
   Since that PR had a very long history of tuning the workflows (sincere thanks to @Akshat-Jain for the assistance!),
   I have created a fresh PR instead to simplify reviews.
   
   The old PR had only a couple of comments:
   https://github.com/apache/druid/pull/18265#discussion_r2214466052
   https://github.com/apache/druid/pull/18265#discussion_r2214471841
   
   which have already been answered.
   
   ### Changes
   
   - Add extension `druid-testcontainers` that can be later published as an [individual module](https://testcontainers.com/modules/) for `Testcontainers`
   - Add `DruidContainer` - `Testcontainer` impl for running individual Druid services
   - Add `DruidContainerResource` to allow use of `DruidContainer` in embedded cluster tests
   - Make minor modifications to `EmbeddedDruidServer` and `EmbeddedDruidCluster` to support
   running both embedded servers and container-based services in the same cluster
   - Add job to run the `*DockerTest`s (2 right now, this number should remain small over time).
   
   Note: __Usage of `DruidContainer` will not be the norm but the exception to test out backward compatibility
   and/or Docker related changes only. All other use cases should continue to use `EmbeddedDruidServers`.__
   
   ### Pending items
   
   - Populate all the test methods using the builders in #18207 once it is merged
   
   ### Advantages
   
   - Can be used to run embedded tests against the distribution Docker image as opposed to the current ITs
   which use a custom IT-only image
   - Takes us 1 step closer to completely removing the old IT frameworks
   - Leverages all the benefits of the embedded test framework while also testing out the Docker containers
   - Can be easily used for backward compatibility tests
   
   ### Next steps
   
   - Once we have phased out the old ITs, all integration tests will be based on `EmbeddedClusterTestBase`
   - Most of these tests will just use `EmbeddedDruidServer` (and required containers only, like `KafkaResource`)
   - These tests will use Indexer as worker as they are faster and lighter than MiddleManagers
   - There will be only a small number of tests that use `DruidContainer`, some of which will use MiddleManagers
   
   <hr>
   
   This PR has:
   
   - [ ] been self-reviewed.
      - [ ] using the [concurrency checklist](https://github.com/apache/druid/blob/master/dev/code-review/concurrency.md) (Remove this item if the PR doesn't have any relation to concurrency.)
   - [ ] added documentation for new or modified features or behaviors.
   - [ ] a release note entry in the PR description.
   - [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
   - [ ] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/dev/license.md)
   - [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
   - [ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
   - [ ] added integration tests.
   - [ ] been tested in a test Druid cluster.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "capistrant (via GitHub)" <gi...@apache.org>.

capistrant commented on code in PR #18302:
URL: https://github.com/apache/druid/pull/18302#discussion_r2255284939


##########
server/src/test/java/org/apache/druid/server/metrics/LatchableEmitter.java:
##########
@@ -253,7 +255,7 @@ public EventMatcher hasMetricName(String metricName)
     /**
      * Matches an event only if it has the given metric value.

Review Comment:
   nit
   
   ```suggestion
        * Matches an event only if it has a value equal to or greater than the given metric value.
   ```



##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/psql/PostgreSQLMetadataStoreTest.java:
##########
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.psql;
+
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.EmbeddedRouter;
+import org.apache.druid.testing.embedded.indexing.IndexTaskTest;
+
+/**
+ * Same as {@link IndexTaskTest}, but using a MariaDB metadata store instead of Derby.

Review Comment:
   nit
   
   ```suggestion
    * Same as {@link IndexTaskTest}, but using a PostgreSQL metadata store instead of Derby.
   ```



##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/psql/PostgreSQLMetadataResource.java:
##########
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.psql;
+
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.metadata.storage.postgresql.PostgreSQLMetadataStorageModule;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.TestcontainerResource;
+import org.testcontainers.containers.PostgreSQLContainer;
+
+/**
+ * Resource that creates a MariaDB metadata store.

Review Comment:
   nit
   
   ```suggestion
    * Resource that creates a PostgreSQL metadata store.
   ```



##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/CustomNodeRoleDockerTest.java:
##########
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import org.apache.druid.testing.DruidCommand;
+import org.apache.druid.testing.cli.CliEventCollector;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.EmbeddedRouter;
+import org.apache.druid.testing.embedded.server.HighAvailabilityTest;
+import org.junit.jupiter.api.Test;
+
+import java.util.Map;
+
+public class CustomNodeRoleDockerTest extends DockerTestBase

Review Comment:
   nit: short javadoc relaying what we are testing and why it is valuable test will be helpful for future folks checking in on this



##########
services/src/test/java/org/apache/druid/testing/embedded/EmbeddedDruidCluster.java:
##########
@@ -232,13 +228,21 @@ public TestFolder getTestFolder()
   }
 
   /**
-   * The embedded Zookeeper server used by this cluster, if any.
-   *
-   * @throws NullPointerException if this cluster has no embedded zookeeper.
+   * Uses a container-friendly hostname for all embedded services, Druid as well
+   * as external.
+   */
+  public EmbeddedDruidCluster useContainerFriendlyHostname()

Review Comment:
   what is the reason of having clusters opt into containerFriendlyHostname? Is there a downside or problem of using it if you don't have to, say for an existing embedded test you have written, HighAvailability?



##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/IngestionBackwardCompatibilityDockerTest.java:
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.druid.client.coordinator.CoordinatorServiceClient;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.http.client.response.BytesFullResponseHandler;
+import org.apache.druid.java.util.http.client.response.BytesFullResponseHolder;
+import org.apache.druid.rpc.RequestBuilder;
+import org.apache.druid.rpc.indexing.SegmentUpdateResponse;
+import org.apache.druid.testing.DruidContainer;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.jboss.netty.handler.codec.http.HttpMethod;
+import org.junit.jupiter.api.Nested;
+
+/**
+ * Runs some basic ingestion tests using {@code DruidContainers} to verify
+ * backward compatibility with old Druid images.
+ */
+public class IngestionBackwardCompatibilityDockerTest
+{
+  @Nested
+  public class Apache31 extends IngestionDockerTest
+  {
+    @Override
+    public EmbeddedDruidCluster createCluster()
+    {
+      coordinator.usingImage(DruidContainer.Image.APACHE_31);

Review Comment:
   is this something someone is going to have to periodically update and commit to master?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "gianm (via GitHub)" <gi...@apache.org>.

gianm commented on code in PR #18302:
URL: https://github.com/apache/druid/pull/18302#discussion_r2238749731


##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/IngestionDockerTest.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.druid.common.utils.IdUtils;
+import org.apache.druid.data.input.impl.CsvInputFormat;
+import org.apache.druid.data.input.impl.DimensionsSpec;
+import org.apache.druid.data.input.impl.TimestampSpec;
+import org.apache.druid.indexer.TaskState;
+import org.apache.druid.indexer.TaskStatusPlus;
+import org.apache.druid.indexing.kafka.KafkaIndexTaskModule;
+import org.apache.druid.indexing.kafka.simulate.KafkaResource;
+import org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorIOConfig;
+import org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorSpec;
+import org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorTuningConfig;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.supervisor.SupervisorStatus;
+import org.apache.druid.java.util.common.DateTimes;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.parsers.CloseableIterator;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.indexing.DataSchema;
+import org.apache.druid.testing.embedded.EmbeddedBroker;
+import org.apache.druid.testing.embedded.EmbeddedClusterApis;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.EmbeddedOverlord;
+import org.apache.druid.testing.embedded.EmbeddedRouter;
+import org.apache.druid.testing.embedded.derby.EmbeddedDerbyMetadataResource;
+import org.apache.druid.testing.embedded.indexing.Resources;
+import org.apache.druid.testing.embedded.junit5.EmbeddedClusterTestBase;
+import org.apache.druid.testing.embedded.minio.MinIOStorageResource;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.joda.time.DateTime;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ThreadLocalRandom;
+
+/**
+ * Runs some basic ingestion tests using {@link DruidContainers}.
+ */
+public class IngestionDockerTest extends EmbeddedClusterTestBase
+{
+  static {
+    System.setProperty("druid.testing.docker.image", "apache/druid:35.0.0-SNAPSHOT");
+  }
+
+  // Druid Docker containers
+  protected final DruidContainerResource overlordLeader = DruidContainers.newOverlord().withTestImage();
+  protected final DruidContainerResource coordinator = DruidContainers.newCoordinator().withTestImage();
+  protected final DruidContainerResource historical = DruidContainers.newHistorical().withTestImage();
+  protected final DruidContainerResource broker1 = DruidContainers.newBroker().withTestImage();
+  protected final DruidContainerResource middleManager = DruidContainers
+      .newMiddleManager()
+      .withTestImage()
+      .addProperty("druid.segment.handoff.pollDuration", "PT0.1s");
+
+  // Follower EmbeddedOverlord to watch segment publish events
+  private final EmbeddedOverlord overlordFollower = new EmbeddedOverlord()
+      .addProperty("druid.plaintextPort", "7090")
+      .addProperty("druid.manager.segments.useIncrementalCache", "always")
+      .addProperty("druid.manager.segments.pollDuration", "PT0.1s");
+
+  // Additional EmbeddedBroker to wait for segments to become queryable
+  private final EmbeddedBroker broker2 = new EmbeddedBroker()
+      .addProperty("druid.plaintextPort", "7082");
+
+  private final KafkaResource kafkaServer = new KafkaResource();
+
+  @Override
+  public EmbeddedDruidCluster createCluster()
+  {
+    final EmbeddedDruidCluster cluster = EmbeddedDruidCluster.withZookeeper();
+
+    return cluster
+        .useDruidContainers()
+        .useLatchableEmitter()
+        // Needed for overlordFollower to recognize the KafkaSupervisor type
+        .addExtension(KafkaIndexTaskModule.class)
+        .addResource(new EmbeddedDerbyMetadataResource())
+        .addResource(new MinIOStorageResource())
+        .addResource(kafkaServer)
+        .addCommonProperty(
+            "druid.extensions.loadList",
+            "[\"druid-s3-extensions\", \"druid-kafka-indexing-service\", \"druid-multi-stage-query\"]"
+        )
+        .addResource(coordinator)
+        .addResource(overlordLeader)
+        .addResource(middleManager)
+        .addResource(historical)
+        .addResource(broker1)
+        .addServer(overlordFollower)
+        .addServer(broker2)
+        .addServer(new EmbeddedRouter());
+  }
+
+  @Test
+  public void test_runIndexTask_andKillData()
+  {
+
+  }
+
+  @Test
+  public void test_runIndexParallelTask_andCompactData()
+  {
+
+  }
+
+  @Test
+  public void test_runMsqTask_andQueryData()
+  {
+
+  }
+
+  @Test
+  public void test_runIndexTask()
+  {
+    final String taskId = IdUtils.getRandomId();
+    final Object task = createIndexTaskForInlineData(
+        taskId,
+        StringUtils.replace(Resources.CSV_DATA_10_DAYS, "\n", "\\n")
+    );
+
+    cluster.callApi().onLeaderOverlord(o -> o.runTask(taskId, task));
+    waitForCachedUsedSegmentCount(10);
+    verifyUsedSegmentCount(10);
+  }
+
+  private Object createIndexTaskForInlineData(String taskId, String inlineDataCsv)
+  {
+    return EmbeddedClusterApis.createTaskFromPayload(
+        taskId,
+        StringUtils.format(Resources.INDEX_TASK_PAYLOAD_WITH_INLINE_DATA, inlineDataCsv, dataSource)
+    );
+  }
+
+  @Test
+  public void test_runKafkaSupervisor()
+  {
+    final String topic = dataSource;
+    kafkaServer.createTopicWithPartitions(topic, 2);
+
+    kafkaServer.produceRecordsToTopic(
+        generateRecordsForTopic(topic, 10, DateTimes.of("2025-06-01"))
+    );
+
+    // Submit and start a supervisor
+    final String supervisorId = dataSource;
+    final KafkaSupervisorSpec kafkaSupervisorSpec = createKafkaSupervisor(supervisorId, topic);
+
+    final Map<String, String> startSupervisorResult = cluster.callApi().onLeaderOverlord(
+        o -> o.postSupervisor(kafkaSupervisorSpec)
+    );
+    Assertions.assertEquals(Map.of("id", supervisorId), startSupervisorResult);
+
+    // Wait for the broker to discover the realtime segments
+    broker2.latchableEmitter().waitForEvent(
+        event -> event.hasDimension(DruidMetrics.DATASOURCE, dataSource)
+    );
+
+    SupervisorStatus supervisorStatus = cluster.callApi().getSupervisorStatus(supervisorId);
+    Assertions.assertFalse(supervisorStatus.isSuspended());
+    Assertions.assertTrue(supervisorStatus.isHealthy());
+    Assertions.assertEquals("RUNNING", supervisorStatus.getState());
+    Assertions.assertEquals(topic, supervisorStatus.getSource());
+
+    // Get the task statuses
+    List<TaskStatusPlus> taskStatuses = ImmutableList.copyOf(
+        (CloseableIterator<TaskStatusPlus>)
+            cluster.callApi().onLeaderOverlord(o -> o.taskStatuses(null, dataSource, 1))
+    );
+    Assertions.assertEquals(1, taskStatuses.size());
+    Assertions.assertEquals(TaskState.RUNNING, taskStatuses.get(0).getStatusCode());
+
+    // Verify the count of rows ingested into the datasource so far
+    // Assertions.assertEquals("10", cluster.runSql("SELECT COUNT(*) FROM %s", dataSource));

Review Comment:
   Please don't commit commented-out code.



##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/IngestionDockerTest.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.druid.common.utils.IdUtils;
+import org.apache.druid.data.input.impl.CsvInputFormat;
+import org.apache.druid.data.input.impl.DimensionsSpec;
+import org.apache.druid.data.input.impl.TimestampSpec;
+import org.apache.druid.indexer.TaskState;
+import org.apache.druid.indexer.TaskStatusPlus;
+import org.apache.druid.indexing.kafka.KafkaIndexTaskModule;
+import org.apache.druid.indexing.kafka.simulate.KafkaResource;
+import org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorIOConfig;
+import org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorSpec;
+import org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorTuningConfig;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.supervisor.SupervisorStatus;
+import org.apache.druid.java.util.common.DateTimes;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.parsers.CloseableIterator;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.indexing.DataSchema;
+import org.apache.druid.testing.embedded.EmbeddedBroker;
+import org.apache.druid.testing.embedded.EmbeddedClusterApis;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.EmbeddedOverlord;
+import org.apache.druid.testing.embedded.EmbeddedRouter;
+import org.apache.druid.testing.embedded.derby.EmbeddedDerbyMetadataResource;
+import org.apache.druid.testing.embedded.indexing.Resources;
+import org.apache.druid.testing.embedded.junit5.EmbeddedClusterTestBase;
+import org.apache.druid.testing.embedded.minio.MinIOStorageResource;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.joda.time.DateTime;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ThreadLocalRandom;
+
+/**
+ * Runs some basic ingestion tests using {@link DruidContainers}.
+ */
+public class IngestionDockerTest extends EmbeddedClusterTestBase
+{
+  static {
+    System.setProperty("druid.testing.docker.image", "apache/druid:35.0.0-SNAPSHOT");

Review Comment:
   Setting this in a static initializer doesn't seem right, because it will be set when the class is loaded, not necessarily when the test runs. Maybe with `@BeforeAll`.
   
   Although, it'd be even better to not have to set the property at all. Like, perhaps the property just sets a default, but tests are able to specify their own when constructing the `DruidContainer`. That's how it works for other kinds of testcontainers.
   
   Also- about the value- do we need to change the value when Druid's version is upgraded? Is it possible to get it programmatically, such as from `DruidContainerResource.class.getPackage().getImplementationVersion()`? Should work when running from a jar, possibly not from an IDE.
   
   Finally, if we are going to keep needing to set this property, better to refer to the name by `DruidContainerResource.PROPERTY_TEST_IMAGE` rather than hardcoding it.



##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/IngestionDockerTest.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.druid.common.utils.IdUtils;
+import org.apache.druid.data.input.impl.CsvInputFormat;
+import org.apache.druid.data.input.impl.DimensionsSpec;
+import org.apache.druid.data.input.impl.TimestampSpec;
+import org.apache.druid.indexer.TaskState;
+import org.apache.druid.indexer.TaskStatusPlus;
+import org.apache.druid.indexing.kafka.KafkaIndexTaskModule;
+import org.apache.druid.indexing.kafka.simulate.KafkaResource;
+import org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorIOConfig;
+import org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorSpec;
+import org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorTuningConfig;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.supervisor.SupervisorStatus;
+import org.apache.druid.java.util.common.DateTimes;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.parsers.CloseableIterator;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.indexing.DataSchema;
+import org.apache.druid.testing.embedded.EmbeddedBroker;
+import org.apache.druid.testing.embedded.EmbeddedClusterApis;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.EmbeddedOverlord;
+import org.apache.druid.testing.embedded.EmbeddedRouter;
+import org.apache.druid.testing.embedded.derby.EmbeddedDerbyMetadataResource;
+import org.apache.druid.testing.embedded.indexing.Resources;
+import org.apache.druid.testing.embedded.junit5.EmbeddedClusterTestBase;
+import org.apache.druid.testing.embedded.minio.MinIOStorageResource;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.joda.time.DateTime;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ThreadLocalRandom;
+
+/**
+ * Runs some basic ingestion tests using {@link DruidContainers}.
+ */
+public class IngestionDockerTest extends EmbeddedClusterTestBase
+{
+  static {
+    System.setProperty("druid.testing.docker.image", "apache/druid:35.0.0-SNAPSHOT");
+  }
+
+  // Druid Docker containers
+  protected final DruidContainerResource overlordLeader = DruidContainers.newOverlord().withTestImage();
+  protected final DruidContainerResource coordinator = DruidContainers.newCoordinator().withTestImage();
+  protected final DruidContainerResource historical = DruidContainers.newHistorical().withTestImage();
+  protected final DruidContainerResource broker1 = DruidContainers.newBroker().withTestImage();
+  protected final DruidContainerResource middleManager = DruidContainers
+      .newMiddleManager()
+      .withTestImage()
+      .addProperty("druid.segment.handoff.pollDuration", "PT0.1s");
+
+  // Follower EmbeddedOverlord to watch segment publish events
+  private final EmbeddedOverlord overlordFollower = new EmbeddedOverlord()
+      .addProperty("druid.plaintextPort", "7090")
+      .addProperty("druid.manager.segments.useIncrementalCache", "always")
+      .addProperty("druid.manager.segments.pollDuration", "PT0.1s");
+
+  // Additional EmbeddedBroker to wait for segments to become queryable
+  private final EmbeddedBroker broker2 = new EmbeddedBroker()
+      .addProperty("druid.plaintextPort", "7082");
+
+  private final KafkaResource kafkaServer = new KafkaResource();
+
+  @Override
+  public EmbeddedDruidCluster createCluster()
+  {
+    final EmbeddedDruidCluster cluster = EmbeddedDruidCluster.withZookeeper();
+
+    return cluster
+        .useDruidContainers()
+        .useLatchableEmitter()
+        // Needed for overlordFollower to recognize the KafkaSupervisor type
+        .addExtension(KafkaIndexTaskModule.class)
+        .addResource(new EmbeddedDerbyMetadataResource())
+        .addResource(new MinIOStorageResource())
+        .addResource(kafkaServer)
+        .addCommonProperty(
+            "druid.extensions.loadList",
+            "[\"druid-s3-extensions\", \"druid-kafka-indexing-service\", \"druid-multi-stage-query\"]"
+        )
+        .addResource(coordinator)
+        .addResource(overlordLeader)
+        .addResource(middleManager)
+        .addResource(historical)
+        .addResource(broker1)
+        .addServer(overlordFollower)
+        .addServer(broker2)
+        .addServer(new EmbeddedRouter());
+  }
+
+  @Test
+  public void test_runIndexTask_andKillData()
+  {
+

Review Comment:
   a comment about why this is empty would be helpful (same goes for the others)



##########
extensions-core/druid-testcontainers/pom.xml:
##########
@@ -0,0 +1,70 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+  ~ Licensed to the Apache Software Foundation (ASF) under one
+  ~ or more contributor license agreements.  See the NOTICE file
+  ~ distributed with this work for additional information
+  ~ regarding copyright ownership.  The ASF licenses this file
+  ~ to you under the Apache License, Version 2.0 (the
+  ~ "License"); you may not use this file except in compliance
+  ~ with the License.  You may obtain a copy of the License at
+  ~
+  ~   http://www.apache.org/licenses/LICENSE-2.0
+  ~
+  ~ Unless required by applicable law or agreed to in writing,
+  ~ software distributed under the License is distributed on an
+  ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  ~ KIND, either express or implied.  See the License for the
+  ~ specific language governing permissions and limitations
+  ~ under the License.
+  -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
+  <modelVersion>4.0.0</modelVersion>
+
+  <groupId>org.apache.druid.extensions</groupId>
+  <artifactId>druid-testcontainers</artifactId>
+  <name>druid-testcontainers</name>
+  <description>Testcontainers for running Apache Druid services in tests</description>
+
+  <parent>
+    <groupId>org.apache.druid</groupId>
+    <artifactId>druid</artifactId>
+    <version>35.0.0-SNAPSHOT</version>
+    <relativePath>../../pom.xml</relativePath>
+  </parent>
+
+  <!-- This module should have minimal dependencies so that it can be used independently
+   by libraries and codebases outside Druid to run Testcontainers -->
+  <dependencies>
+    <dependency>
+      <groupId>org.apache.druid</groupId>
+      <artifactId>druid-processing</artifactId>

Review Comment:
   `druid-processing` is a pretty big dependency with lots of transitive dependencies. It looks like it's just for `StringUtils` and `Logger`. It would be better to avoid those and remove this dependency. For `Logger`, you could either refrain from logging stuff, or you could use `org.slf4j.Logger` directly and depend on `slf4j-api`.



##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/DruidContainerResource.java:
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import org.apache.druid.java.util.common.FileUtils;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.testing.DruidCommand;
+import org.apache.druid.testing.DruidContainer;
+import org.apache.druid.testing.MountedDir;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.TestcontainerResource;
+import org.testcontainers.utility.DockerImageName;
+
+import java.io.File;
+import java.nio.file.Files;
+import java.nio.file.attribute.PosixFilePermissions;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * {@link TestcontainerResource} to run Druid services.
+ * Currently, only core extensions can be used out-of-the-box with these containers
+ * such as {@code druid-s3-extensions} or {@code postgresql-metadata-storage},
+ * simply by adding them to {@code druid.extensions.loadList}.
+ * <p>
+ * {@link DruidContainers} should be used only for testing backward compatiblity
+ * or a Docker-specific feature. For all other testing needs, use plain old
+ * {@code EmbeddedDruidServer} as they are much faster, allow easy debugging and
+ * do not require downloading any images.
+ */
+public class DruidContainerResource extends TestcontainerResource<DruidContainer>
+{
+  /**
+   * Java system property to specify the name of the Docker test image.
+   */
+  public static final String PROPERTY_TEST_IMAGE = "druid.testing.docker.image";
+
+  private static final Logger log = new Logger(DruidContainerResource.class);
+
+  /**
+   * Forbidden server properties that may be used by EmbeddedDruidServers but
+   * interfere with the functioning of DruidContainer-based services.
+   */
+  private static final Set<String> FORBIDDEN_PROPERTIES = Set.of(
+      "druid.extensions.modulesForEmbeddedTests",
+      "druid.emitter"
+  );
+
+  /**
+   * A static incremental ID is used instead of a random number to ensure that
+   * tests are more deterministic and easier to debug.
+   */
+  private static final AtomicInteger SERVER_ID = new AtomicInteger(0);
+
+  private final String name;
+  private final DruidCommand command;
+  private final Map<String, String> properties = new HashMap<>();
+
+  private DockerImageName imageName;
+  private EmbeddedDruidCluster cluster;
+
+  private File containerDirectory;
+
+  private MountedDir indexerLogsDeepStorageDirectory;
+  private MountedDir serviceLogsDirectory;
+  private MountedDir segmentDeepStorageDirectory;
+
+  DruidContainerResource(DruidCommand command)
+  {
+    this.name = StringUtils.format(
+        "container_%s_%d",
+        command.getName(),
+        SERVER_ID.incrementAndGet()
+    );
+    this.command = command;
+    addProperty("druid.host", EmbeddedDruidCluster.getDefaultHost());
+  }
+
+  public DruidContainerResource withImage(DockerImageName imageName)
+  {
+    this.imageName = imageName;
+    return this;
+  }
+
+  /**
+   * Uses the Docker test image specified by the system property
+   * {@link #PROPERTY_TEST_IMAGE} for this container.
+   */
+  public DruidContainerResource withTestImage()

Review Comment:
   Similar comment as above, IMO `usingTestImage()` is clearer.



##########
services/src/test/java/org/apache/druid/testing/embedded/EmbeddedDruidCluster.java:
##########
@@ -231,13 +242,50 @@ public TestFolder getTestFolder()
   }
 
   /**
-   * The embedded Zookeeper server used by this cluster, if any.
-   *
-   * @throws NullPointerException if this cluster has no embedded zookeeper.
+   * Hostname to be used for embedded services (both Druid or external).
+   * Using this hostname ensures that the underlying service is reachable by both
+   * EmbeddedDruidServers and DruidContainers.
+   */
+  public String getEmbeddedServiceHostname()
+  {
+    return hasDruidContainers ? getDefaultHost() : "localhost";
+  }
+
+  /**
+   * Hostname for the host machine running the containers. Using this hostname
+   * instead of "localhost" allows all the Druid containers to talk to each
+   * other and also other EmbeddedDruidServers.
    */
-  public EmbeddedZookeeper getZookeeper()
+  public static String getDefaultHost()
   {
-    return Objects.requireNonNull(zookeeper, "No embedded zookeeper configured for this cluster");
+    try {
+      return InetAddress.getLocalHost().getHostAddress();
+    }
+    catch (UnknownHostException e) {
+      throw new ISE(e, "Unable to determine host name");
+    }
+  }
+
+  /**
+   * Replaces {@code localhost} or {@code 127.0.0.1} in the given connectUri
+   * with {@link #getEmbeddedServiceHostname()}. Using the embedded URI ensures
+   * that the underlying service is reachable by both EmbeddedDruidServers and
+   * DruidContainers.
+   */
+  public String getEmbeddedConnectUri(String connectUri)
+  {
+    if (!hasDruidContainers) {
+      return connectUri;
+    } else if (connectUri.contains("localhost")) {
+      return StringUtils.replace(connectUri, "localhost", getEmbeddedServiceHostname());

Review Comment:
   Seeing URI manipulations like this stresses me out ;P
   
   It's just test code, but it's still better form to manipulate the URL properly. That means parsing it with `new URI(connectUri)` and then creating a replacement `URI` using the 7-arg constructor.



##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/IngestionDockerTest.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.druid.common.utils.IdUtils;
+import org.apache.druid.data.input.impl.CsvInputFormat;
+import org.apache.druid.data.input.impl.DimensionsSpec;
+import org.apache.druid.data.input.impl.TimestampSpec;
+import org.apache.druid.indexer.TaskState;
+import org.apache.druid.indexer.TaskStatusPlus;
+import org.apache.druid.indexing.kafka.KafkaIndexTaskModule;
+import org.apache.druid.indexing.kafka.simulate.KafkaResource;
+import org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorIOConfig;
+import org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorSpec;
+import org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorTuningConfig;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.supervisor.SupervisorStatus;
+import org.apache.druid.java.util.common.DateTimes;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.parsers.CloseableIterator;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.indexing.DataSchema;
+import org.apache.druid.testing.embedded.EmbeddedBroker;
+import org.apache.druid.testing.embedded.EmbeddedClusterApis;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.EmbeddedOverlord;
+import org.apache.druid.testing.embedded.EmbeddedRouter;
+import org.apache.druid.testing.embedded.derby.EmbeddedDerbyMetadataResource;
+import org.apache.druid.testing.embedded.indexing.Resources;
+import org.apache.druid.testing.embedded.junit5.EmbeddedClusterTestBase;
+import org.apache.druid.testing.embedded.minio.MinIOStorageResource;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.joda.time.DateTime;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ThreadLocalRandom;
+
+/**
+ * Runs some basic ingestion tests using {@link DruidContainers}.
+ */
+public class IngestionDockerTest extends EmbeddedClusterTestBase
+{
+  static {
+    System.setProperty("druid.testing.docker.image", "apache/druid:35.0.0-SNAPSHOT");
+  }
+
+  // Druid Docker containers
+  protected final DruidContainerResource overlordLeader = DruidContainers.newOverlord().withTestImage();

Review Comment:
   I don't think we can be sure that this one is always the leader. It starts up first, but maybe something happens that causes a leader change. Or maybe for some reason the container version fails to gain leadership due to an error, and then the `overlordFollower` becomes leader.
   
   It would help to add some check in the tests that confirms this one is actually leading. Ideally we check at the both beginning and end of each test.



##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/DruidContainerResource.java:
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import org.apache.druid.java.util.common.FileUtils;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.testing.DruidCommand;
+import org.apache.druid.testing.DruidContainer;
+import org.apache.druid.testing.MountedDir;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.TestcontainerResource;
+import org.testcontainers.utility.DockerImageName;
+
+import java.io.File;
+import java.nio.file.Files;
+import java.nio.file.attribute.PosixFilePermissions;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * {@link TestcontainerResource} to run Druid services.
+ * Currently, only core extensions can be used out-of-the-box with these containers
+ * such as {@code druid-s3-extensions} or {@code postgresql-metadata-storage},
+ * simply by adding them to {@code druid.extensions.loadList}.
+ * <p>
+ * {@link DruidContainers} should be used only for testing backward compatiblity
+ * or a Docker-specific feature. For all other testing needs, use plain old
+ * {@code EmbeddedDruidServer} as they are much faster, allow easy debugging and
+ * do not require downloading any images.
+ */
+public class DruidContainerResource extends TestcontainerResource<DruidContainer>
+{
+  /**
+   * Java system property to specify the name of the Docker test image.
+   */
+  public static final String PROPERTY_TEST_IMAGE = "druid.testing.docker.image";
+
+  private static final Logger log = new Logger(DruidContainerResource.class);
+
+  /**
+   * Forbidden server properties that may be used by EmbeddedDruidServers but
+   * interfere with the functioning of DruidContainer-based services.
+   */
+  private static final Set<String> FORBIDDEN_PROPERTIES = Set.of(
+      "druid.extensions.modulesForEmbeddedTests",
+      "druid.emitter"
+  );
+
+  /**
+   * A static incremental ID is used instead of a random number to ensure that
+   * tests are more deterministic and easier to debug.
+   */
+  private static final AtomicInteger SERVER_ID = new AtomicInteger(0);
+
+  private final String name;
+  private final DruidCommand command;
+  private final Map<String, String> properties = new HashMap<>();
+
+  private DockerImageName imageName;
+  private EmbeddedDruidCluster cluster;
+
+  private File containerDirectory;
+
+  private MountedDir indexerLogsDeepStorageDirectory;
+  private MountedDir serviceLogsDirectory;
+  private MountedDir segmentDeepStorageDirectory;
+
+  DruidContainerResource(DruidCommand command)
+  {
+    this.name = StringUtils.format(
+        "container_%s_%d",
+        command.getName(),
+        SERVER_ID.incrementAndGet()
+    );
+    this.command = command;
+    addProperty("druid.host", EmbeddedDruidCluster.getDefaultHost());
+  }
+
+  public DruidContainerResource withImage(DockerImageName imageName)

Review Comment:
   To me a method named `withX` suggests a copy is being made, and the original object is not modified. But here the original object is in fact modified. IMO `setImage` or `useImage` would be clearer.



##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/IngestionDockerTest.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.druid.common.utils.IdUtils;
+import org.apache.druid.data.input.impl.CsvInputFormat;
+import org.apache.druid.data.input.impl.DimensionsSpec;
+import org.apache.druid.data.input.impl.TimestampSpec;
+import org.apache.druid.indexer.TaskState;
+import org.apache.druid.indexer.TaskStatusPlus;
+import org.apache.druid.indexing.kafka.KafkaIndexTaskModule;
+import org.apache.druid.indexing.kafka.simulate.KafkaResource;
+import org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorIOConfig;
+import org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorSpec;
+import org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorTuningConfig;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.supervisor.SupervisorStatus;
+import org.apache.druid.java.util.common.DateTimes;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.parsers.CloseableIterator;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.indexing.DataSchema;
+import org.apache.druid.testing.embedded.EmbeddedBroker;
+import org.apache.druid.testing.embedded.EmbeddedClusterApis;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.EmbeddedOverlord;
+import org.apache.druid.testing.embedded.EmbeddedRouter;
+import org.apache.druid.testing.embedded.derby.EmbeddedDerbyMetadataResource;
+import org.apache.druid.testing.embedded.indexing.Resources;
+import org.apache.druid.testing.embedded.junit5.EmbeddedClusterTestBase;
+import org.apache.druid.testing.embedded.minio.MinIOStorageResource;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.joda.time.DateTime;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ThreadLocalRandom;
+
+/**
+ * Runs some basic ingestion tests using {@link DruidContainers}.
+ */
+public class IngestionDockerTest extends EmbeddedClusterTestBase
+{
+  static {
+    System.setProperty("druid.testing.docker.image", "apache/druid:35.0.0-SNAPSHOT");
+  }
+
+  // Druid Docker containers
+  protected final DruidContainerResource overlordLeader = DruidContainers.newOverlord().withTestImage();
+  protected final DruidContainerResource coordinator = DruidContainers.newCoordinator().withTestImage();
+  protected final DruidContainerResource historical = DruidContainers.newHistorical().withTestImage();
+  protected final DruidContainerResource broker1 = DruidContainers.newBroker().withTestImage();
+  protected final DruidContainerResource middleManager = DruidContainers
+      .newMiddleManager()
+      .withTestImage()
+      .addProperty("druid.segment.handoff.pollDuration", "PT0.1s");
+
+  // Follower EmbeddedOverlord to watch segment publish events
+  private final EmbeddedOverlord overlordFollower = new EmbeddedOverlord()
+      .addProperty("druid.plaintextPort", "7090")
+      .addProperty("druid.manager.segments.useIncrementalCache", "always")
+      .addProperty("druid.manager.segments.pollDuration", "PT0.1s");
+
+  // Additional EmbeddedBroker to wait for segments to become queryable
+  private final EmbeddedBroker broker2 = new EmbeddedBroker()
+      .addProperty("druid.plaintextPort", "7082");
+
+  private final KafkaResource kafkaServer = new KafkaResource();
+
+  @Override
+  public EmbeddedDruidCluster createCluster()
+  {
+    final EmbeddedDruidCluster cluster = EmbeddedDruidCluster.withZookeeper();
+
+    return cluster
+        .useDruidContainers()
+        .useLatchableEmitter()
+        // Needed for overlordFollower to recognize the KafkaSupervisor type
+        .addExtension(KafkaIndexTaskModule.class)
+        .addResource(new EmbeddedDerbyMetadataResource())
+        .addResource(new MinIOStorageResource())
+        .addResource(kafkaServer)
+        .addCommonProperty(
+            "druid.extensions.loadList",
+            "[\"druid-s3-extensions\", \"druid-kafka-indexing-service\", \"druid-multi-stage-query\"]"
+        )
+        .addResource(coordinator)
+        .addResource(overlordLeader)
+        .addResource(middleManager)
+        .addResource(historical)
+        .addResource(broker1)
+        .addServer(overlordFollower)

Review Comment:
   a comment about why we mix containers and noncontainers in this test would be helpful.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "gianm (via GitHub)" <gi...@apache.org>.

gianm commented on code in PR #18302:
URL: https://github.com/apache/druid/pull/18302#discussion_r2238718755


##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/IngestionDockerTest.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.druid.common.utils.IdUtils;
+import org.apache.druid.data.input.impl.CsvInputFormat;
+import org.apache.druid.data.input.impl.DimensionsSpec;
+import org.apache.druid.data.input.impl.TimestampSpec;
+import org.apache.druid.indexer.TaskState;
+import org.apache.druid.indexer.TaskStatusPlus;
+import org.apache.druid.indexing.kafka.KafkaIndexTaskModule;
+import org.apache.druid.indexing.kafka.simulate.KafkaResource;
+import org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorIOConfig;
+import org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorSpec;
+import org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorTuningConfig;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.supervisor.SupervisorStatus;
+import org.apache.druid.java.util.common.DateTimes;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.parsers.CloseableIterator;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.indexing.DataSchema;
+import org.apache.druid.testing.embedded.EmbeddedBroker;
+import org.apache.druid.testing.embedded.EmbeddedClusterApis;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.EmbeddedOverlord;
+import org.apache.druid.testing.embedded.EmbeddedRouter;
+import org.apache.druid.testing.embedded.derby.EmbeddedDerbyMetadataResource;
+import org.apache.druid.testing.embedded.indexing.Resources;
+import org.apache.druid.testing.embedded.junit5.EmbeddedClusterTestBase;
+import org.apache.druid.testing.embedded.minio.MinIOStorageResource;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.joda.time.DateTime;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ThreadLocalRandom;
+
+/**
+ * Runs some basic ingestion tests using {@link DruidContainers}.
+ */
+public class IngestionDockerTest extends EmbeddedClusterTestBase
+{
+  static {
+    System.setProperty("druid.testing.docker.image", "apache/druid:35.0.0-SNAPSHOT");

Review Comment:
   Setting this in a static initializer doesn't seem right, because it will be set when the class is loaded, not necessarily when the test runs. Maybe better to use `@BeforeAll`.
   
   Although, it'd be even better to not have to set the property at all. Like, perhaps the property just sets a default, but tests are able to specify their own when constructing the `DruidContainer`. That's how it works for other kinds of testcontainers.
   
   Also- about the value- do we need to change the value when Druid's version is upgraded? Is it possible to get it programmatically, such as from `DruidContainerResource.class.getPackage().getImplementationVersion()`? Should work when running from a jar, possibly not from an IDE.
   
   Finally, if we are going to keep needing to set this property, better to refer to the name by `DruidContainerResource.PROPERTY_TEST_IMAGE` rather than hardcoding it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.

kfaraz commented on code in PR #18302:
URL: https://github.com/apache/druid/pull/18302#discussion_r2244409048


##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/IngestionDockerTest.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.druid.common.utils.IdUtils;
+import org.apache.druid.data.input.impl.CsvInputFormat;
+import org.apache.druid.data.input.impl.DimensionsSpec;
+import org.apache.druid.data.input.impl.TimestampSpec;
+import org.apache.druid.indexer.TaskState;
+import org.apache.druid.indexer.TaskStatusPlus;
+import org.apache.druid.indexing.kafka.KafkaIndexTaskModule;
+import org.apache.druid.indexing.kafka.simulate.KafkaResource;
+import org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorIOConfig;
+import org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorSpec;
+import org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorTuningConfig;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.supervisor.SupervisorStatus;
+import org.apache.druid.java.util.common.DateTimes;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.parsers.CloseableIterator;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.indexing.DataSchema;
+import org.apache.druid.testing.embedded.EmbeddedBroker;
+import org.apache.druid.testing.embedded.EmbeddedClusterApis;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.EmbeddedOverlord;
+import org.apache.druid.testing.embedded.EmbeddedRouter;
+import org.apache.druid.testing.embedded.derby.EmbeddedDerbyMetadataResource;
+import org.apache.druid.testing.embedded.indexing.Resources;
+import org.apache.druid.testing.embedded.junit5.EmbeddedClusterTestBase;
+import org.apache.druid.testing.embedded.minio.MinIOStorageResource;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.joda.time.DateTime;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ThreadLocalRandom;
+
+/**
+ * Runs some basic ingestion tests using {@link DruidContainers}.
+ */
+public class IngestionDockerTest extends EmbeddedClusterTestBase
+{
+  static {
+    System.setProperty("druid.testing.docker.image", "apache/druid:35.0.0-SNAPSHOT");

Review Comment:
   Thanks for bringing this up, @gianm !
   
   For release voting, I feel it would be best for committers to download the image built as part of the RC and then running `mvn test -Ddruid.testing.docker.image=apache/druid:rc1 -Dtest=*DockerTest*`.
   Since the release voting process is not automated anyway (perhaps by design so that voters are aware of the steps they go through), it is probably okay to require specifying `-Ddruid.testing.docker.image` explicitly.
   
   We can probably add these instructions to release guidelines (and the vote mail) so that users can always copy paste the commands.
   
   What are your thoughts?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.

kfaraz commented on code in PR #18302:
URL: https://github.com/apache/druid/pull/18302#discussion_r2236980099


##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/DruidContainerResource.java:
##########
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import org.apache.druid.java.util.common.FileUtils;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.testing.DruidCommand;
+import org.apache.druid.testing.DruidContainer;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.TestcontainerResource;
+import org.testcontainers.containers.BindMode;
+import org.testcontainers.containers.wait.strategy.Wait;
+import org.testcontainers.utility.DockerImageName;
+
+import java.io.File;
+import java.nio.file.Files;
+import java.nio.file.attribute.PosixFilePermissions;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+
+/**
+ * {@link TestcontainerResource} to run Druid services.
+ * Currently, only core extensions can be used out-of-the-box with these containers
+ * such as {@code druid-s3-extensions} or {@code postgresql-metadata-storage},
+ * simply by adding them to {@code druid.extensions.loadList}.
+ * <p>
+ * {@link DruidContainers} should be used only for testing backward compatiblity
+ * or a Docker-specific feature. For all other testing needs, use plain old
+ * {@code EmbeddedDruidServer} as they are much faster, allow easy debugging and
+ * do not require downloading any images.
+ */
+public class DruidContainerResource extends TestcontainerResource<DruidContainer>
+{
+  /**
+   * Java system property to specify the name of the Docker test image.
+   */
+  public static final String PROPERTY_TEST_IMAGE = "druid.testing.docker.image";
+
+  private static final Logger log = new Logger(DruidContainerResource.class);
+
+  /**
+   * Forbidden server properties that may be used by EmbeddedDruidServers but
+   * interfere with the functioning of DruidContainer-based services.
+   */
+  private static final Set<String> FORBIDDEN_PROPERTIES = Set.of(
+      "druid.extensions.modulesForEmbeddedTests",
+      "druid.emitter"
+  );
+
+  /**
+   * A static incremental ID is used instead of a random number to ensure that
+   * tests are more deterministic and easier to debug.
+   */
+  private static final AtomicInteger SERVER_ID = new AtomicInteger(0);
+
+  private final String name;
+  private final DruidCommand command;
+  private final Map<String, String> properties = new HashMap<>();
+  private final List<Integer> taskPorts = new ArrayList<>();
+
+  private int servicePort;
+  private DockerImageName imageName;
+  private EmbeddedDruidCluster cluster;
+
+  private String containerDirectory;
+
+  private MountedDir indexerLogsDeepStorageDirectory;
+  private MountedDir serviceLogsDirectory;
+  private MountedDir segmentDeepStorageDirectory;
+
+  DruidContainerResource(DruidCommand command)
+  {
+    this.name = StringUtils.format(
+        "container_%s_%d",
+        command.getName(),
+        SERVER_ID.incrementAndGet()
+    );
+    this.command = command;
+
+    Integer[] exposedPorts = command.getExposedPorts();
+    servicePort = exposedPorts[0];
+    taskPorts.addAll(Arrays.asList(exposedPorts).subList(1, exposedPorts.length));
+  }
+
+  public DruidContainerResource withPort(int port)
+  {
+    this.servicePort = port;
+    return this;
+  }
+
+  /**
+   * Used to bind the task ports on middle managers.
+   */
+  public DruidContainerResource withTaskPorts(int startPort, int workerCapacity)
+  {
+    taskPorts.clear();
+    for (int i = 0; i < workerCapacity; ++i) {
+      taskPorts.add(startPort + i);
+    }
+    addProperty("druid.worker.capacity", String.valueOf(workerCapacity));
+    addProperty("druid.indexer.runner.startPort", String.valueOf(startPort));
+    addProperty("druid.indexer.runner.endPort", String.valueOf(startPort + workerCapacity));
+
+    return this;
+  }
+
+  public DruidContainerResource withImage(DockerImageName imageName)
+  {
+    this.imageName = imageName;
+    return this;
+  }
+
+  /**
+   * Uses the Docker test image specified by the system property
+   * {@link #PROPERTY_TEST_IMAGE} for this container.
+   */
+  public DruidContainerResource withTestImage()
+  {
+    String imageName = Objects.requireNonNull(
+        System.getProperty(PROPERTY_TEST_IMAGE),
+        StringUtils.format("System property[%s] is not set", PROPERTY_TEST_IMAGE)
+    );
+    return withImage(DockerImageName.parse(imageName));
+  }
+
+  public DruidContainerResource addProperty(String key, String value)
+  {
+    properties.put(key, value);
+    return this;
+  }
+
+  @Override
+  public void beforeStart(EmbeddedDruidCluster cluster)
+  {
+    this.cluster = cluster;
+
+    // Mount directories used by the entire cluster (including embedded servers)
+    this.segmentDeepStorageDirectory = new MountedDir(
+        "/tmp/druid/deep-store",
+        cluster.getTestFolder().getOrCreateFolder("deep-store").getAbsolutePath()
+    );
+    this.indexerLogsDeepStorageDirectory = new MountedDir(
+        "/tmp/druid/indexer-logs",
+        cluster.getTestFolder().getOrCreateFolder("indexer-logs").getAbsolutePath()
+    );
+
+    // Mount directories used by this container for easier debugging with service logs
+    this.containerDirectory = cluster.getTestFolder().getOrCreateFolder(name).getAbsolutePath();
+    this.serviceLogsDirectory = new MountedDir(
+        "/opt/druid/log",
+        containerDirectory + "/log"
+    );
+
+    // Create the log directory upfront to avoid permission issues
+    createLogDirectory(new File(containerDirectory, "log"));
+  }
+
+  @Override
+  protected DruidContainer createContainer()
+  {
+    final List<Integer> exposedPorts = new ArrayList<>(taskPorts);
+    exposedPorts.add(servicePort);
+    log.info(
+        "Starting Druid container[%s] on port[%d] with mounted directory[%s] and exposed ports[%s].",
+        name, servicePort, containerDirectory, exposedPorts
+    );
+
+    final DruidContainer container = new DruidContainer(command, imageName)
+        .withExposedPorts(exposedPorts.toArray(new Integer[0]))
+        .withCommonProperties(getCommonProperties())
+        .withServiceProperties(getServerProperties())
+        .withFileSystemBind(
+            segmentDeepStorageDirectory.hostPath,
+            segmentDeepStorageDirectory.containerPath,
+            BindMode.READ_WRITE
+        )
+        .withFileSystemBind(
+            indexerLogsDeepStorageDirectory.hostPath,
+            indexerLogsDeepStorageDirectory.containerPath,
+            BindMode.READ_WRITE
+        )
+        .withFileSystemBind(
+            serviceLogsDirectory.hostPath,
+            serviceLogsDirectory.containerPath,
+            BindMode.READ_WRITE
+        )
+        .withEnv(
+            Map.of(
+                "DRUID_SET_HOST_IP", "0",
+                "DRUID_SET_HOST", "0"
+            )
+        )
+        .waitingFor(Wait.forHttp("/status/health").forPort(servicePort));
+
+    // Bind the ports statically (rather than using a mapped port) to the same
+    // value used in `druid.plaintextPort` to make this node discoverable
+    // by other services (both embedded and dockerized).
+    List<String> portBindings = exposedPorts.stream().map(
+        port -> StringUtils.format("%d:%d", port, port)
+    ).collect(Collectors.toList());
+    container.setPortBindings(portBindings);
+
+    return container;
+  }
+
+  /**
+   * Writes the common properties to a file.
+   *
+   * @return Name of the file.
+   */
+  private Properties getCommonProperties()

Review Comment:
   fixed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "Akshat-Jain (via GitHub)" <gi...@apache.org>.

Akshat-Jain closed pull request #18302: Add DruidContainers to run docker tests with embedded-test framework
URL: https://github.com/apache/druid/pull/18302


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "Akshat-Jain (via GitHub)" <gi...@apache.org>.

Akshat-Jain closed pull request #18302: Add DruidContainers to run docker tests with embedded-test framework
URL: https://github.com/apache/druid/pull/18302


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "kgyrtkirk (via GitHub)" <gi...@apache.org>.

kgyrtkirk commented on code in PR #18302:
URL: https://github.com/apache/druid/pull/18302#discussion_r2237203178


##########
.github/workflows/docker-tests.yml:
##########
@@ -0,0 +1,97 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: "Docker Tests using Distribution Image"
+on:
+  workflow_call:
+
+jobs:
+  run-docker-tests:
+    name: Run Docker tests
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Retrieve Docker container
+        id: docker-restore
+        uses: actions/cache/restore@v4
+        with:
+          key: druid-dist-container.tar.gz-${{ github.sha }}
+          path: |
+            ./druid-dist-container.tar.gz
+      - name: Stop and remove Druid Docker containers
+        run: |
+          echo "Force stopping all Druid containers and pruning"
+          docker ps -aq --filter "ancestor=apache/druid" | xargs -r docker rm -f
+          docker system prune -af --volumes
+      - name: Load Docker image
+        run: |
+          docker load --input druid-dist-container.tar.gz
+          docker images
+      - name: Setup Java
+        uses: actions/setup-java@v4
+        with:
+          distribution: 'zulu'
+          java-version: 17
+          cache: 'maven'
+      - name: Run *DockerTest*
+        id: run-it
+        run: |
+          source .github/scripts/distribution_checks_env.sh
+          .github/scripts/run_unit-tests -Dtest=*DockerTest* -Ddruid.testing.docker.image=$DRUID_DIST_IMAGE_NAME

Review Comment:
   you could run it locally to reproduce what's happening on the CI; you could call `mvn` directly; but you will also need to build the image...



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.

kfaraz commented on code in PR #18302:
URL: https://github.com/apache/druid/pull/18302#discussion_r2239050224


##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/DruidContainerResource.java:
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import org.apache.druid.java.util.common.FileUtils;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.testing.DruidCommand;
+import org.apache.druid.testing.DruidContainer;
+import org.apache.druid.testing.MountedDir;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.TestcontainerResource;
+import org.testcontainers.utility.DockerImageName;
+
+import java.io.File;
+import java.nio.file.Files;
+import java.nio.file.attribute.PosixFilePermissions;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * {@link TestcontainerResource} to run Druid services.
+ * Currently, only core extensions can be used out-of-the-box with these containers
+ * such as {@code druid-s3-extensions} or {@code postgresql-metadata-storage},
+ * simply by adding them to {@code druid.extensions.loadList}.
+ * <p>
+ * {@link DruidContainers} should be used only for testing backward compatiblity
+ * or a Docker-specific feature. For all other testing needs, use plain old
+ * {@code EmbeddedDruidServer} as they are much faster, allow easy debugging and
+ * do not require downloading any images.
+ */
+public class DruidContainerResource extends TestcontainerResource<DruidContainer>
+{
+  /**
+   * Java system property to specify the name of the Docker test image.
+   */
+  public static final String PROPERTY_TEST_IMAGE = "druid.testing.docker.image";
+
+  private static final Logger log = new Logger(DruidContainerResource.class);
+
+  /**
+   * Forbidden server properties that may be used by EmbeddedDruidServers but
+   * interfere with the functioning of DruidContainer-based services.
+   */
+  private static final Set<String> FORBIDDEN_PROPERTIES = Set.of(
+      "druid.extensions.modulesForEmbeddedTests",
+      "druid.emitter"
+  );
+
+  /**
+   * A static incremental ID is used instead of a random number to ensure that
+   * tests are more deterministic and easier to debug.
+   */
+  private static final AtomicInteger SERVER_ID = new AtomicInteger(0);
+
+  private final String name;
+  private final DruidCommand command;
+  private final Map<String, String> properties = new HashMap<>();
+
+  private DockerImageName imageName;
+  private EmbeddedDruidCluster cluster;
+
+  private File containerDirectory;
+
+  private MountedDir indexerLogsDeepStorageDirectory;
+  private MountedDir serviceLogsDirectory;
+  private MountedDir segmentDeepStorageDirectory;
+
+  DruidContainerResource(DruidCommand command)
+  {
+    this.name = StringUtils.format(
+        "container_%s_%d",
+        command.getName(),
+        SERVER_ID.incrementAndGet()
+    );
+    this.command = command;
+    addProperty("druid.host", EmbeddedDruidCluster.getDefaultHost());
+  }
+
+  public DruidContainerResource withImage(DockerImageName imageName)
+  {
+    this.imageName = imageName;
+    return this;
+  }
+
+  /**
+   * Uses the Docker test image specified by the system property
+   * {@link #PROPERTY_TEST_IMAGE} for this container.
+   */
+  public DruidContainerResource withTestImage()

Review Comment:
   Fixed.



##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/DruidContainerResource.java:
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import org.apache.druid.java.util.common.FileUtils;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.testing.DruidCommand;
+import org.apache.druid.testing.DruidContainer;
+import org.apache.druid.testing.MountedDir;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.TestcontainerResource;
+import org.testcontainers.utility.DockerImageName;
+
+import java.io.File;
+import java.nio.file.Files;
+import java.nio.file.attribute.PosixFilePermissions;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * {@link TestcontainerResource} to run Druid services.
+ * Currently, only core extensions can be used out-of-the-box with these containers
+ * such as {@code druid-s3-extensions} or {@code postgresql-metadata-storage},
+ * simply by adding them to {@code druid.extensions.loadList}.
+ * <p>
+ * {@link DruidContainers} should be used only for testing backward compatiblity
+ * or a Docker-specific feature. For all other testing needs, use plain old
+ * {@code EmbeddedDruidServer} as they are much faster, allow easy debugging and
+ * do not require downloading any images.
+ */
+public class DruidContainerResource extends TestcontainerResource<DruidContainer>
+{
+  /**
+   * Java system property to specify the name of the Docker test image.
+   */
+  public static final String PROPERTY_TEST_IMAGE = "druid.testing.docker.image";
+
+  private static final Logger log = new Logger(DruidContainerResource.class);
+
+  /**
+   * Forbidden server properties that may be used by EmbeddedDruidServers but
+   * interfere with the functioning of DruidContainer-based services.
+   */
+  private static final Set<String> FORBIDDEN_PROPERTIES = Set.of(
+      "druid.extensions.modulesForEmbeddedTests",
+      "druid.emitter"
+  );
+
+  /**
+   * A static incremental ID is used instead of a random number to ensure that
+   * tests are more deterministic and easier to debug.
+   */
+  private static final AtomicInteger SERVER_ID = new AtomicInteger(0);
+
+  private final String name;
+  private final DruidCommand command;
+  private final Map<String, String> properties = new HashMap<>();
+
+  private DockerImageName imageName;
+  private EmbeddedDruidCluster cluster;
+
+  private File containerDirectory;
+
+  private MountedDir indexerLogsDeepStorageDirectory;
+  private MountedDir serviceLogsDirectory;
+  private MountedDir segmentDeepStorageDirectory;
+
+  DruidContainerResource(DruidCommand command)
+  {
+    this.name = StringUtils.format(
+        "container_%s_%d",
+        command.getName(),
+        SERVER_ID.incrementAndGet()
+    );
+    this.command = command;
+    addProperty("druid.host", EmbeddedDruidCluster.getDefaultHost());
+  }
+
+  public DruidContainerResource withImage(DockerImageName imageName)

Review Comment:
   fixed, thanks for the suggestion!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "gianm (via GitHub)" <gi...@apache.org>.

gianm commented on code in PR #18302:
URL: https://github.com/apache/druid/pull/18302#discussion_r2259443909


##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/IngestionBackwardCompatibilityDockerTest.java:
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.druid.client.coordinator.CoordinatorServiceClient;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.http.client.response.BytesFullResponseHandler;
+import org.apache.druid.java.util.http.client.response.BytesFullResponseHolder;
+import org.apache.druid.rpc.RequestBuilder;
+import org.apache.druid.rpc.indexing.SegmentUpdateResponse;
+import org.apache.druid.testing.DruidContainer;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.jboss.netty.handler.codec.http.HttpMethod;
+import org.junit.jupiter.api.Nested;
+
+/**
+ * Runs some basic ingestion tests using {@code DruidContainers} to verify
+ * backward compatibility with old Druid images.
+ */
+public class IngestionBackwardCompatibilityDockerTest
+{
+  @Nested
+  public class Apache31 extends IngestionDockerTest
+  {
+    @Override
+    public EmbeddedDruidCluster createCluster()
+    {
+      coordinator.usingImage(DruidContainer.Image.APACHE_31);

Review Comment:
   I think this test will get skipped by default since it has `DockerTest` in the name. IMO, it's sensible to always run this particular test. The main reason I thought that Docker tests in general shouldn't all run is that they're mostly intended to run against the "current code". They don't serve their purpose unless the person running the tests actually creates an image matching the current checked out repo. This needs to be done manually so it makes sense that it's a separate command.
   
   But, the backwards compatibility test isn't intended to deploy the "current code". It wants to deploy a specific older version. So, it doesn't have that problem, and it's therefore good to always run it.
   
   Btw, IMO it's nicer to use annotations rather than test names to control what tests run in a standard mvn run. The downside of using names is that it's indirect. It makes it less obvious which tests run and which don't, and also means that if you want to change whether a test runs, you have to rename it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.

kfaraz merged PR #18302:
URL: https://github.com/apache/druid/pull/18302


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.

kfaraz commented on code in PR #18302:
URL: https://github.com/apache/druid/pull/18302#discussion_r2236985621


##########
.github/workflows/docker-tests.yml:
##########
@@ -0,0 +1,97 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: "Docker Tests using Distribution Image"
+on:
+  workflow_call:
+
+jobs:
+  run-docker-tests:
+    name: Run Docker tests
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Retrieve Docker container
+        id: docker-restore
+        uses: actions/cache/restore@v4
+        with:
+          key: druid-dist-container.tar.gz-${{ github.sha }}
+          path: |
+            ./druid-dist-container.tar.gz
+      - name: Stop and remove Druid Docker containers
+        run: |
+          echo "Force stopping all Druid containers and pruning"
+          docker ps -aq --filter "ancestor=apache/druid" | xargs -r docker rm -f
+          docker system prune -af --volumes
+      - name: Load Docker image
+        run: |
+          docker load --input druid-dist-container.tar.gz
+          docker images
+      - name: Setup Java
+        uses: actions/setup-java@v4
+        with:
+          distribution: 'zulu'
+          java-version: 17
+          cache: 'maven'
+      - name: Run *DockerTest*
+        id: run-it
+        run: |
+          source .github/scripts/distribution_checks_env.sh
+          .github/scripts/run_unit-tests -Dtest=*DockerTest* -Ddruid.testing.docker.image=$DRUID_DIST_IMAGE_NAME
+        timeout-minutes: 60
+
+      - name: Collect docker logs on failure
+        if: ${{ failure() && steps.run-it.conclusion == 'failure' }}
+        run: |
+          mkdir docker-logs
+          for c in $(docker ps -a --format="{{.Names}}")
+          do
+            docker logs $c > ./docker-logs/$c.log
+          done
+
+      - name: Tar docker logs
+        if: ${{ failure() && steps.run-it.conclusion == 'failure' }}
+        run: tar cvzf ./docker-logs.tgz ./docker-logs
+
+      - name: Upload docker logs to GitHub
+        if: ${{ failure() && steps.run-it.conclusion == 'failure' }}
+        uses: actions/upload-artifact@v4
+        with:
+          name: failure-docker-logs
+          path: docker-logs.tgz
+
+      - name: Collect service logs on failure
+        if: ${{ failure() && steps.run-it.conclusion == 'failure' }}
+        run: |
+          tar cvzf ./service-logs.tgz /tmp/embedded-tests

Review Comment:
   removed this step.



##########
.github/workflows/distribution-checks.yml:
##########
@@ -38,4 +38,23 @@ jobs:
     steps:
       - uses: actions/checkout@v4
       - name: Build the Docker image
-        run: DOCKER_BUILDKIT=1 docker build -t apache/druid:tag -f distribution/docker/Dockerfile .
+        run: |

Review Comment:
   fixed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "Akshat-Jain (via GitHub)" <gi...@apache.org>.

Akshat-Jain closed pull request #18302: Add DruidContainers to run docker tests with embedded-test framework
URL: https://github.com/apache/druid/pull/18302


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "gianm (via GitHub)" <gi...@apache.org>.

gianm commented on code in PR #18302:
URL: https://github.com/apache/druid/pull/18302#discussion_r2239424762


##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/IngestionDockerTest.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.druid.common.utils.IdUtils;
+import org.apache.druid.data.input.impl.CsvInputFormat;
+import org.apache.druid.data.input.impl.DimensionsSpec;
+import org.apache.druid.data.input.impl.TimestampSpec;
+import org.apache.druid.indexer.TaskState;
+import org.apache.druid.indexer.TaskStatusPlus;
+import org.apache.druid.indexing.kafka.KafkaIndexTaskModule;
+import org.apache.druid.indexing.kafka.simulate.KafkaResource;
+import org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorIOConfig;
+import org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorSpec;
+import org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorTuningConfig;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.supervisor.SupervisorStatus;
+import org.apache.druid.java.util.common.DateTimes;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.parsers.CloseableIterator;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.indexing.DataSchema;
+import org.apache.druid.testing.embedded.EmbeddedBroker;
+import org.apache.druid.testing.embedded.EmbeddedClusterApis;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.EmbeddedOverlord;
+import org.apache.druid.testing.embedded.EmbeddedRouter;
+import org.apache.druid.testing.embedded.derby.EmbeddedDerbyMetadataResource;
+import org.apache.druid.testing.embedded.indexing.Resources;
+import org.apache.druid.testing.embedded.junit5.EmbeddedClusterTestBase;
+import org.apache.druid.testing.embedded.minio.MinIOStorageResource;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.joda.time.DateTime;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ThreadLocalRandom;
+
+/**
+ * Runs some basic ingestion tests using {@link DruidContainers}.
+ */
+public class IngestionDockerTest extends EmbeddedClusterTestBase
+{
+  static {
+    System.setProperty("druid.testing.docker.image", "apache/druid:35.0.0-SNAPSHOT");

Review Comment:
   I think it's fine to require people to set the system property for local testing. Some thought would be needed on part of the person running the tests about what image they want to use. If they want to test against their current branch, they'll need to build a Docker image. Maybe they don't care about that; maybe they just want to test the Docker embedded test itself, in which case it would be fine to run against a released Apache image.
   
   The error message when the property isn't set should spell this out. It should mention that the property is typically set during CI, but isn't set outside CI because we don't know what image you want to use.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "gianm (via GitHub)" <gi...@apache.org>.

gianm commented on code in PR #18302:
URL: https://github.com/apache/druid/pull/18302#discussion_r2239428497


##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/IngestionDockerTest.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.druid.common.utils.IdUtils;
+import org.apache.druid.data.input.impl.CsvInputFormat;
+import org.apache.druid.data.input.impl.DimensionsSpec;
+import org.apache.druid.data.input.impl.TimestampSpec;
+import org.apache.druid.indexer.TaskState;
+import org.apache.druid.indexer.TaskStatusPlus;
+import org.apache.druid.indexing.kafka.KafkaIndexTaskModule;
+import org.apache.druid.indexing.kafka.simulate.KafkaResource;
+import org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorIOConfig;
+import org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorSpec;
+import org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorTuningConfig;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.supervisor.SupervisorStatus;
+import org.apache.druid.java.util.common.DateTimes;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.parsers.CloseableIterator;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.indexing.DataSchema;
+import org.apache.druid.testing.embedded.EmbeddedBroker;
+import org.apache.druid.testing.embedded.EmbeddedClusterApis;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.EmbeddedOverlord;
+import org.apache.druid.testing.embedded.EmbeddedRouter;
+import org.apache.druid.testing.embedded.derby.EmbeddedDerbyMetadataResource;
+import org.apache.druid.testing.embedded.indexing.Resources;
+import org.apache.druid.testing.embedded.junit5.EmbeddedClusterTestBase;
+import org.apache.druid.testing.embedded.minio.MinIOStorageResource;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.joda.time.DateTime;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ThreadLocalRandom;
+
+/**
+ * Runs some basic ingestion tests using {@link DruidContainers}.
+ */
+public class IngestionDockerTest extends EmbeddedClusterTestBase
+{
+  static {
+    System.setProperty("druid.testing.docker.image", "apache/druid:35.0.0-SNAPSHOT");

Review Comment:
   Some thought also needs to be given to how committers should run `mvn test` when they test release builds as part of release voting. Do they need to build a Docker image first in order to run this test? Do they need to specify `-Ddruid.testing.docker.image` to `mvn`? Or is it automatic as part of the `mvn` workflow? I am curious what you think about these items.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "Akshat-Jain (via GitHub)" <gi...@apache.org>.

Akshat-Jain closed pull request #18302: Add DruidContainers to run docker tests with embedded-test framework
URL: https://github.com/apache/druid/pull/18302


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "Akshat-Jain (via GitHub)" <gi...@apache.org>.

Akshat-Jain closed pull request #18302: Add DruidContainers to run docker tests with embedded-test framework
URL: https://github.com/apache/druid/pull/18302


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "kgyrtkirk (via GitHub)" <gi...@apache.org>.

kgyrtkirk commented on code in PR #18302:
URL: https://github.com/apache/druid/pull/18302#discussion_r2235755695


##########
.github/scripts/distribution_checks_env.sh:
##########
@@ -0,0 +1,20 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+export DRUID_DIST_IMAGE_NAME=apache/druid:35.0.0-SNAPSHOT

Review Comment:
   doesn't feel right - you could use the workflow level `env` to declare exports
   
   ....or move the full logic into a script and run that...



##########
.github/workflows/ci.yml:
##########
@@ -27,7 +27,7 @@ jobs:
         pattern: [ "C*", "H*,U*,V*", "N*,Q*,S*", "B*,O*,R*", "G*,J*,K*", "F*,L*,M*", "A*,D*,I*,X*,Y*,Z*", "E*,P*,T*,W*"]
     uses: ./.github/workflows/worker.yml
     with:
-      script: .github/scripts/run_unit-tests -Dtest=!QTest,'${{ matrix.pattern }}' -Dmaven.test.failure.ignore=true
+      script: .github/scripts/run_unit-tests -Dtest=!QTest,!*DockerTest*,'${{ matrix.pattern }}' -Dmaven.test.failure.ignore=true

Review Comment:
   I'm quilty of not moving the `QTest` into the `validation` phase ; which seem to have given rise to this pattern which could be considered bad practice
   
   this is not really serious...it could be fixed separaetly - but nicely outlines your requirement that you just wanted to add 1 more test....



##########
.github/workflows/docker-tests.yml:
##########
@@ -0,0 +1,97 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: "Docker Tests using Distribution Image"
+on:
+  workflow_call:
+
+jobs:
+  run-docker-tests:
+    name: Run Docker tests
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Retrieve Docker container
+        id: docker-restore
+        uses: actions/cache/restore@v4
+        with:
+          key: druid-dist-container.tar.gz-${{ github.sha }}
+          path: |
+            ./druid-dist-container.tar.gz
+      - name: Stop and remove Druid Docker containers
+        run: |
+          echo "Force stopping all Druid containers and pruning"
+          docker ps -aq --filter "ancestor=apache/druid" | xargs -r docker rm -f
+          docker system prune -af --volumes
+      - name: Load Docker image
+        run: |
+          docker load --input druid-dist-container.tar.gz
+          docker images
+      - name: Setup Java
+        uses: actions/setup-java@v4
+        with:
+          distribution: 'zulu'
+          java-version: 17
+          cache: 'maven'
+      - name: Run *DockerTest*
+        id: run-it
+        run: |
+          source .github/scripts/distribution_checks_env.sh
+          .github/scripts/run_unit-tests -Dtest=*DockerTest* -Ddruid.testing.docker.image=$DRUID_DIST_IMAGE_NAME
+        timeout-minutes: 60
+
+      - name: Collect docker logs on failure
+        if: ${{ failure() && steps.run-it.conclusion == 'failure' }}
+        run: |
+          mkdir docker-logs
+          for c in $(docker ps -a --format="{{.Names}}")
+          do
+            docker logs $c > ./docker-logs/$c.log
+          done
+
+      - name: Tar docker logs
+        if: ${{ failure() && steps.run-it.conclusion == 'failure' }}
+        run: tar cvzf ./docker-logs.tgz ./docker-logs
+
+      - name: Upload docker logs to GitHub
+        if: ${{ failure() && steps.run-it.conclusion == 'failure' }}
+        uses: actions/upload-artifact@v4
+        with:
+          name: failure-docker-logs
+          path: docker-logs.tgz
+
+      - name: Collect service logs on failure
+        if: ${{ failure() && steps.run-it.conclusion == 'failure' }}
+        run: |
+          tar cvzf ./service-logs.tgz /tmp/embedded-tests

Review Comment:
   I wonder why these are under `/tmp` and not under a `target` directory? 



##########
.github/workflows/docker-tests.yml:
##########
@@ -0,0 +1,97 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: "Docker Tests using Distribution Image"
+on:
+  workflow_call:
+
+jobs:
+  run-docker-tests:
+    name: Run Docker tests
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Retrieve Docker container
+        id: docker-restore
+        uses: actions/cache/restore@v4
+        with:
+          key: druid-dist-container.tar.gz-${{ github.sha }}
+          path: |
+            ./druid-dist-container.tar.gz
+      - name: Stop and remove Druid Docker containers
+        run: |
+          echo "Force stopping all Druid containers and pruning"
+          docker ps -aq --filter "ancestor=apache/druid" | xargs -r docker rm -f
+          docker system prune -af --volumes
+      - name: Load Docker image
+        run: |
+          docker load --input druid-dist-container.tar.gz
+          docker images
+      - name: Setup Java
+        uses: actions/setup-java@v4
+        with:
+          distribution: 'zulu'
+          java-version: 17
+          cache: 'maven'
+      - name: Run *DockerTest*
+        id: run-it
+        run: |
+          source .github/scripts/distribution_checks_env.sh
+          .github/scripts/run_unit-tests -Dtest=*DockerTest* -Ddruid.testing.docker.image=$DRUID_DIST_IMAGE_NAME

Review Comment:
   instead of trying to reuse the `run_unit-tests` script; you could possibly make a separate one for this; and run that (you could still invode `run_unit-tests` from there if you want) 
   
   you should preferrably also use the `worker.yml` to run it....that way if you want to test the workflow you could just try to run that script on your own machine!
   



##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/DruidContainerResource.java:
##########
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import org.apache.druid.java.util.common.FileUtils;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.testing.DruidCommand;
+import org.apache.druid.testing.DruidContainer;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.TestcontainerResource;
+import org.testcontainers.containers.BindMode;
+import org.testcontainers.containers.wait.strategy.Wait;
+import org.testcontainers.utility.DockerImageName;
+
+import java.io.File;
+import java.nio.file.Files;
+import java.nio.file.attribute.PosixFilePermissions;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+
+/**
+ * {@link TestcontainerResource} to run Druid services.
+ * Currently, only core extensions can be used out-of-the-box with these containers
+ * such as {@code druid-s3-extensions} or {@code postgresql-metadata-storage},
+ * simply by adding them to {@code druid.extensions.loadList}.
+ * <p>
+ * {@link DruidContainers} should be used only for testing backward compatiblity
+ * or a Docker-specific feature. For all other testing needs, use plain old
+ * {@code EmbeddedDruidServer} as they are much faster, allow easy debugging and
+ * do not require downloading any images.
+ */
+public class DruidContainerResource extends TestcontainerResource<DruidContainer>
+{
+  /**
+   * Java system property to specify the name of the Docker test image.
+   */
+  public static final String PROPERTY_TEST_IMAGE = "druid.testing.docker.image";
+
+  private static final Logger log = new Logger(DruidContainerResource.class);
+
+  /**
+   * Forbidden server properties that may be used by EmbeddedDruidServers but
+   * interfere with the functioning of DruidContainer-based services.
+   */
+  private static final Set<String> FORBIDDEN_PROPERTIES = Set.of(
+      "druid.extensions.modulesForEmbeddedTests",
+      "druid.emitter"
+  );
+
+  /**
+   * A static incremental ID is used instead of a random number to ensure that
+   * tests are more deterministic and easier to debug.
+   */
+  private static final AtomicInteger SERVER_ID = new AtomicInteger(0);
+
+  private final String name;
+  private final DruidCommand command;
+  private final Map<String, String> properties = new HashMap<>();
+  private final List<Integer> taskPorts = new ArrayList<>();
+
+  private int servicePort;
+  private DockerImageName imageName;
+  private EmbeddedDruidCluster cluster;
+
+  private String containerDirectory;
+
+  private MountedDir indexerLogsDeepStorageDirectory;
+  private MountedDir serviceLogsDirectory;
+  private MountedDir segmentDeepStorageDirectory;
+
+  DruidContainerResource(DruidCommand command)
+  {
+    this.name = StringUtils.format(
+        "container_%s_%d",
+        command.getName(),
+        SERVER_ID.incrementAndGet()
+    );
+    this.command = command;
+
+    Integer[] exposedPorts = command.getExposedPorts();
+    servicePort = exposedPorts[0];
+    taskPorts.addAll(Arrays.asList(exposedPorts).subList(1, exposedPorts.length));
+  }
+
+  public DruidContainerResource withPort(int port)
+  {
+    this.servicePort = port;
+    return this;
+  }
+
+  /**
+   * Used to bind the task ports on middle managers.
+   */
+  public DruidContainerResource withTaskPorts(int startPort, int workerCapacity)
+  {
+    taskPorts.clear();
+    for (int i = 0; i < workerCapacity; ++i) {
+      taskPorts.add(startPort + i);
+    }
+    addProperty("druid.worker.capacity", String.valueOf(workerCapacity));
+    addProperty("druid.indexer.runner.startPort", String.valueOf(startPort));
+    addProperty("druid.indexer.runner.endPort", String.valueOf(startPort + workerCapacity));
+
+    return this;
+  }
+
+  public DruidContainerResource withImage(DockerImageName imageName)
+  {
+    this.imageName = imageName;
+    return this;
+  }
+
+  /**
+   * Uses the Docker test image specified by the system property
+   * {@link #PROPERTY_TEST_IMAGE} for this container.
+   */
+  public DruidContainerResource withTestImage()
+  {
+    String imageName = Objects.requireNonNull(
+        System.getProperty(PROPERTY_TEST_IMAGE),
+        StringUtils.format("System property[%s] is not set", PROPERTY_TEST_IMAGE)
+    );
+    return withImage(DockerImageName.parse(imageName));
+  }
+
+  public DruidContainerResource addProperty(String key, String value)
+  {
+    properties.put(key, value);
+    return this;
+  }
+
+  @Override
+  public void beforeStart(EmbeddedDruidCluster cluster)
+  {
+    this.cluster = cluster;
+
+    // Mount directories used by the entire cluster (including embedded servers)
+    this.segmentDeepStorageDirectory = new MountedDir(
+        "/tmp/druid/deep-store",
+        cluster.getTestFolder().getOrCreateFolder("deep-store").getAbsolutePath()
+    );
+    this.indexerLogsDeepStorageDirectory = new MountedDir(
+        "/tmp/druid/indexer-logs",
+        cluster.getTestFolder().getOrCreateFolder("indexer-logs").getAbsolutePath()
+    );
+
+    // Mount directories used by this container for easier debugging with service logs
+    this.containerDirectory = cluster.getTestFolder().getOrCreateFolder(name).getAbsolutePath();
+    this.serviceLogsDirectory = new MountedDir(
+        "/opt/druid/log",
+        containerDirectory + "/log"
+    );
+
+    // Create the log directory upfront to avoid permission issues
+    createLogDirectory(new File(containerDirectory, "log"));
+  }
+
+  @Override
+  protected DruidContainer createContainer()
+  {
+    final List<Integer> exposedPorts = new ArrayList<>(taskPorts);
+    exposedPorts.add(servicePort);
+    log.info(
+        "Starting Druid container[%s] on port[%d] with mounted directory[%s] and exposed ports[%s].",
+        name, servicePort, containerDirectory, exposedPorts
+    );
+
+    final DruidContainer container = new DruidContainer(command, imageName)
+        .withExposedPorts(exposedPorts.toArray(new Integer[0]))
+        .withCommonProperties(getCommonProperties())
+        .withServiceProperties(getServerProperties())
+        .withFileSystemBind(
+            segmentDeepStorageDirectory.hostPath,
+            segmentDeepStorageDirectory.containerPath,
+            BindMode.READ_WRITE
+        )

Review Comment:
   you could make the `MountedDir` a first class citizen and not extract its belly...



##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/DruidContainerResource.java:
##########
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import org.apache.druid.java.util.common.FileUtils;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.testing.DruidCommand;
+import org.apache.druid.testing.DruidContainer;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.TestcontainerResource;
+import org.testcontainers.containers.BindMode;
+import org.testcontainers.containers.wait.strategy.Wait;
+import org.testcontainers.utility.DockerImageName;
+
+import java.io.File;
+import java.nio.file.Files;
+import java.nio.file.attribute.PosixFilePermissions;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+
+/**
+ * {@link TestcontainerResource} to run Druid services.
+ * Currently, only core extensions can be used out-of-the-box with these containers
+ * such as {@code druid-s3-extensions} or {@code postgresql-metadata-storage},
+ * simply by adding them to {@code druid.extensions.loadList}.
+ * <p>
+ * {@link DruidContainers} should be used only for testing backward compatiblity
+ * or a Docker-specific feature. For all other testing needs, use plain old
+ * {@code EmbeddedDruidServer} as they are much faster, allow easy debugging and
+ * do not require downloading any images.
+ */
+public class DruidContainerResource extends TestcontainerResource<DruidContainer>
+{
+  /**
+   * Java system property to specify the name of the Docker test image.
+   */
+  public static final String PROPERTY_TEST_IMAGE = "druid.testing.docker.image";
+
+  private static final Logger log = new Logger(DruidContainerResource.class);
+
+  /**
+   * Forbidden server properties that may be used by EmbeddedDruidServers but
+   * interfere with the functioning of DruidContainer-based services.
+   */
+  private static final Set<String> FORBIDDEN_PROPERTIES = Set.of(
+      "druid.extensions.modulesForEmbeddedTests",
+      "druid.emitter"
+  );
+
+  /**
+   * A static incremental ID is used instead of a random number to ensure that
+   * tests are more deterministic and easier to debug.
+   */
+  private static final AtomicInteger SERVER_ID = new AtomicInteger(0);
+
+  private final String name;
+  private final DruidCommand command;
+  private final Map<String, String> properties = new HashMap<>();
+  private final List<Integer> taskPorts = new ArrayList<>();
+
+  private int servicePort;
+  private DockerImageName imageName;
+  private EmbeddedDruidCluster cluster;
+
+  private String containerDirectory;
+
+  private MountedDir indexerLogsDeepStorageDirectory;
+  private MountedDir serviceLogsDirectory;
+  private MountedDir segmentDeepStorageDirectory;
+
+  DruidContainerResource(DruidCommand command)
+  {
+    this.name = StringUtils.format(
+        "container_%s_%d",
+        command.getName(),
+        SERVER_ID.incrementAndGet()
+    );
+    this.command = command;
+
+    Integer[] exposedPorts = command.getExposedPorts();
+    servicePort = exposedPorts[0];
+    taskPorts.addAll(Arrays.asList(exposedPorts).subList(1, exposedPorts.length));
+  }
+
+  public DruidContainerResource withPort(int port)
+  {
+    this.servicePort = port;
+    return this;
+  }
+
+  /**
+   * Used to bind the task ports on middle managers.
+   */
+  public DruidContainerResource withTaskPorts(int startPort, int workerCapacity)
+  {
+    taskPorts.clear();
+    for (int i = 0; i < workerCapacity; ++i) {
+      taskPorts.add(startPort + i);
+    }
+    addProperty("druid.worker.capacity", String.valueOf(workerCapacity));
+    addProperty("druid.indexer.runner.startPort", String.valueOf(startPort));
+    addProperty("druid.indexer.runner.endPort", String.valueOf(startPort + workerCapacity));
+
+    return this;
+  }
+
+  public DruidContainerResource withImage(DockerImageName imageName)
+  {
+    this.imageName = imageName;
+    return this;
+  }
+
+  /**
+   * Uses the Docker test image specified by the system property
+   * {@link #PROPERTY_TEST_IMAGE} for this container.
+   */
+  public DruidContainerResource withTestImage()
+  {
+    String imageName = Objects.requireNonNull(
+        System.getProperty(PROPERTY_TEST_IMAGE),
+        StringUtils.format("System property[%s] is not set", PROPERTY_TEST_IMAGE)
+    );
+    return withImage(DockerImageName.parse(imageName));
+  }
+
+  public DruidContainerResource addProperty(String key, String value)
+  {
+    properties.put(key, value);
+    return this;
+  }
+
+  @Override
+  public void beforeStart(EmbeddedDruidCluster cluster)
+  {
+    this.cluster = cluster;
+
+    // Mount directories used by the entire cluster (including embedded servers)
+    this.segmentDeepStorageDirectory = new MountedDir(
+        "/tmp/druid/deep-store",
+        cluster.getTestFolder().getOrCreateFolder("deep-store").getAbsolutePath()
+    );
+    this.indexerLogsDeepStorageDirectory = new MountedDir(
+        "/tmp/druid/indexer-logs",
+        cluster.getTestFolder().getOrCreateFolder("indexer-logs").getAbsolutePath()
+    );
+
+    // Mount directories used by this container for easier debugging with service logs
+    this.containerDirectory = cluster.getTestFolder().getOrCreateFolder(name).getAbsolutePath();
+    this.serviceLogsDirectory = new MountedDir(
+        "/opt/druid/log",
+        containerDirectory + "/log"
+    );
+
+    // Create the log directory upfront to avoid permission issues
+    createLogDirectory(new File(containerDirectory, "log"));
+  }
+
+  @Override
+  protected DruidContainer createContainer()
+  {
+    final List<Integer> exposedPorts = new ArrayList<>(taskPorts);
+    exposedPorts.add(servicePort);
+    log.info(
+        "Starting Druid container[%s] on port[%d] with mounted directory[%s] and exposed ports[%s].",
+        name, servicePort, containerDirectory, exposedPorts
+    );
+
+    final DruidContainer container = new DruidContainer(command, imageName)
+        .withExposedPorts(exposedPorts.toArray(new Integer[0]))
+        .withCommonProperties(getCommonProperties())
+        .withServiceProperties(getServerProperties())
+        .withFileSystemBind(
+            segmentDeepStorageDirectory.hostPath,
+            segmentDeepStorageDirectory.containerPath,
+            BindMode.READ_WRITE
+        )
+        .withFileSystemBind(
+            indexerLogsDeepStorageDirectory.hostPath,
+            indexerLogsDeepStorageDirectory.containerPath,
+            BindMode.READ_WRITE
+        )
+        .withFileSystemBind(
+            serviceLogsDirectory.hostPath,
+            serviceLogsDirectory.containerPath,
+            BindMode.READ_WRITE
+        )
+        .withEnv(
+            Map.of(
+                "DRUID_SET_HOST_IP", "0",
+                "DRUID_SET_HOST", "0"
+            )
+        )
+        .waitingFor(Wait.forHttp("/status/health").forPort(servicePort));
+
+    // Bind the ports statically (rather than using a mapped port) to the same
+    // value used in `druid.plaintextPort` to make this node discoverable
+    // by other services (both embedded and dockerized).
+    List<String> portBindings = exposedPorts.stream().map(
+        port -> StringUtils.format("%d:%d", port, port)
+    ).collect(Collectors.toList());
+    container.setPortBindings(portBindings);
+
+    return container;
+  }
+
+  /**
+   * Writes the common properties to a file.
+   *
+   * @return Name of the file.
+   */
+  private Properties getCommonProperties()

Review Comment:
   methodname and apidoc mismatch



##########
.github/workflows/docker-tests.yml:
##########
@@ -0,0 +1,97 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: "Docker Tests using Distribution Image"
+on:
+  workflow_call:
+
+jobs:
+  run-docker-tests:
+    name: Run Docker tests
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Retrieve Docker container
+        id: docker-restore
+        uses: actions/cache/restore@v4
+        with:
+          key: druid-dist-container.tar.gz-${{ github.sha }}
+          path: |
+            ./druid-dist-container.tar.gz
+      - name: Stop and remove Druid Docker containers
+        run: |
+          echo "Force stopping all Druid containers and pruning"
+          docker ps -aq --filter "ancestor=apache/druid" | xargs -r docker rm -f
+          docker system prune -af --volumes
+      - name: Load Docker image
+        run: |
+          docker load --input druid-dist-container.tar.gz
+          docker images
+      - name: Setup Java
+        uses: actions/setup-java@v4
+        with:
+          distribution: 'zulu'
+          java-version: 17
+          cache: 'maven'
+      - name: Run *DockerTest*
+        id: run-it
+        run: |
+          source .github/scripts/distribution_checks_env.sh
+          .github/scripts/run_unit-tests -Dtest=*DockerTest* -Ddruid.testing.docker.image=$DRUID_DIST_IMAGE_NAME
+        timeout-minutes: 60
+
+      - name: Collect docker logs on failure
+        if: ${{ failure() && steps.run-it.conclusion == 'failure' }}
+        run: |
+          mkdir docker-logs
+          for c in $(docker ps -a --format="{{.Names}}")
+          do
+            docker logs $c > ./docker-logs/$c.log
+          done

Review Comment:
   can you integrate this into the `worker.yml` somehow: put it inside a script and configure the artifact collector to save that as well..
   
   



##########
.github/workflows/distribution-checks.yml:
##########
@@ -38,4 +38,23 @@ jobs:
     steps:
       - uses: actions/checkout@v4
       - name: Build the Docker image
-        run: DOCKER_BUILDKIT=1 docker build -t apache/druid:tag -f distribution/docker/Dockerfile .
+        run: |

Review Comment:
   I feel like what you are doing in this PR is superior to what this workflow was ever doing.
   
   I think we could just remove this whole workflow...along with the `save/load` of the image and run those dockerized tests right after the image is built!



##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/DruidContainerResource.java:
##########
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import org.apache.druid.java.util.common.FileUtils;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.testing.DruidCommand;
+import org.apache.druid.testing.DruidContainer;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.TestcontainerResource;
+import org.testcontainers.containers.BindMode;
+import org.testcontainers.containers.wait.strategy.Wait;
+import org.testcontainers.utility.DockerImageName;
+
+import java.io.File;
+import java.nio.file.Files;
+import java.nio.file.attribute.PosixFilePermissions;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+
+/**
+ * {@link TestcontainerResource} to run Druid services.
+ * Currently, only core extensions can be used out-of-the-box with these containers
+ * such as {@code druid-s3-extensions} or {@code postgresql-metadata-storage},
+ * simply by adding them to {@code druid.extensions.loadList}.
+ * <p>
+ * {@link DruidContainers} should be used only for testing backward compatiblity
+ * or a Docker-specific feature. For all other testing needs, use plain old
+ * {@code EmbeddedDruidServer} as they are much faster, allow easy debugging and
+ * do not require downloading any images.
+ */
+public class DruidContainerResource extends TestcontainerResource<DruidContainer>
+{
+  /**
+   * Java system property to specify the name of the Docker test image.
+   */
+  public static final String PROPERTY_TEST_IMAGE = "druid.testing.docker.image";
+
+  private static final Logger log = new Logger(DruidContainerResource.class);
+
+  /**
+   * Forbidden server properties that may be used by EmbeddedDruidServers but
+   * interfere with the functioning of DruidContainer-based services.
+   */
+  private static final Set<String> FORBIDDEN_PROPERTIES = Set.of(
+      "druid.extensions.modulesForEmbeddedTests",
+      "druid.emitter"
+  );
+
+  /**
+   * A static incremental ID is used instead of a random number to ensure that
+   * tests are more deterministic and easier to debug.
+   */
+  private static final AtomicInteger SERVER_ID = new AtomicInteger(0);
+
+  private final String name;
+  private final DruidCommand command;
+  private final Map<String, String> properties = new HashMap<>();
+  private final List<Integer> taskPorts = new ArrayList<>();
+
+  private int servicePort;
+  private DockerImageName imageName;
+  private EmbeddedDruidCluster cluster;
+
+  private String containerDirectory;

Review Comment:
   its a directory - why it needs to be flattened out to be a `String` ?



##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/DruidContainerResource.java:
##########
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import org.apache.druid.java.util.common.FileUtils;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.testing.DruidCommand;
+import org.apache.druid.testing.DruidContainer;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.TestcontainerResource;
+import org.testcontainers.containers.BindMode;
+import org.testcontainers.containers.wait.strategy.Wait;
+import org.testcontainers.utility.DockerImageName;
+
+import java.io.File;
+import java.nio.file.Files;
+import java.nio.file.attribute.PosixFilePermissions;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+
+/**
+ * {@link TestcontainerResource} to run Druid services.
+ * Currently, only core extensions can be used out-of-the-box with these containers
+ * such as {@code druid-s3-extensions} or {@code postgresql-metadata-storage},
+ * simply by adding them to {@code druid.extensions.loadList}.
+ * <p>
+ * {@link DruidContainers} should be used only for testing backward compatiblity
+ * or a Docker-specific feature. For all other testing needs, use plain old
+ * {@code EmbeddedDruidServer} as they are much faster, allow easy debugging and
+ * do not require downloading any images.
+ */
+public class DruidContainerResource extends TestcontainerResource<DruidContainer>
+{
+  /**
+   * Java system property to specify the name of the Docker test image.
+   */
+  public static final String PROPERTY_TEST_IMAGE = "druid.testing.docker.image";
+
+  private static final Logger log = new Logger(DruidContainerResource.class);
+
+  /**
+   * Forbidden server properties that may be used by EmbeddedDruidServers but
+   * interfere with the functioning of DruidContainer-based services.
+   */
+  private static final Set<String> FORBIDDEN_PROPERTIES = Set.of(
+      "druid.extensions.modulesForEmbeddedTests",
+      "druid.emitter"
+  );
+
+  /**
+   * A static incremental ID is used instead of a random number to ensure that
+   * tests are more deterministic and easier to debug.
+   */
+  private static final AtomicInteger SERVER_ID = new AtomicInteger(0);
+
+  private final String name;
+  private final DruidCommand command;
+  private final Map<String, String> properties = new HashMap<>();
+  private final List<Integer> taskPorts = new ArrayList<>();
+
+  private int servicePort;
+  private DockerImageName imageName;
+  private EmbeddedDruidCluster cluster;
+
+  private String containerDirectory;
+
+  private MountedDir indexerLogsDeepStorageDirectory;
+  private MountedDir serviceLogsDirectory;
+  private MountedDir segmentDeepStorageDirectory;
+
+  DruidContainerResource(DruidCommand command)
+  {
+    this.name = StringUtils.format(
+        "container_%s_%d",

Review Comment:
   not matters much - but I feel like `container_` doesn't really gives a hint what this is...would be better if it would hint that its being created by Druid's embedded test framework....don't have a good suggestion...maybe its not that important
   



##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/DruidContainerResource.java:
##########
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import org.apache.druid.java.util.common.FileUtils;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.testing.DruidCommand;
+import org.apache.druid.testing.DruidContainer;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.TestcontainerResource;
+import org.testcontainers.containers.BindMode;
+import org.testcontainers.containers.wait.strategy.Wait;
+import org.testcontainers.utility.DockerImageName;
+
+import java.io.File;
+import java.nio.file.Files;
+import java.nio.file.attribute.PosixFilePermissions;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+
+/**
+ * {@link TestcontainerResource} to run Druid services.
+ * Currently, only core extensions can be used out-of-the-box with these containers
+ * such as {@code druid-s3-extensions} or {@code postgresql-metadata-storage},
+ * simply by adding them to {@code druid.extensions.loadList}.
+ * <p>
+ * {@link DruidContainers} should be used only for testing backward compatiblity
+ * or a Docker-specific feature. For all other testing needs, use plain old
+ * {@code EmbeddedDruidServer} as they are much faster, allow easy debugging and
+ * do not require downloading any images.
+ */
+public class DruidContainerResource extends TestcontainerResource<DruidContainer>
+{
+  /**
+   * Java system property to specify the name of the Docker test image.
+   */
+  public static final String PROPERTY_TEST_IMAGE = "druid.testing.docker.image";
+
+  private static final Logger log = new Logger(DruidContainerResource.class);
+
+  /**
+   * Forbidden server properties that may be used by EmbeddedDruidServers but
+   * interfere with the functioning of DruidContainer-based services.
+   */
+  private static final Set<String> FORBIDDEN_PROPERTIES = Set.of(
+      "druid.extensions.modulesForEmbeddedTests",
+      "druid.emitter"
+  );
+
+  /**
+   * A static incremental ID is used instead of a random number to ensure that
+   * tests are more deterministic and easier to debug.
+   */
+  private static final AtomicInteger SERVER_ID = new AtomicInteger(0);
+
+  private final String name;
+  private final DruidCommand command;
+  private final Map<String, String> properties = new HashMap<>();
+  private final List<Integer> taskPorts = new ArrayList<>();
+
+  private int servicePort;
+  private DockerImageName imageName;
+  private EmbeddedDruidCluster cluster;
+
+  private String containerDirectory;
+
+  private MountedDir indexerLogsDeepStorageDirectory;
+  private MountedDir serviceLogsDirectory;
+  private MountedDir segmentDeepStorageDirectory;
+
+  DruidContainerResource(DruidCommand command)
+  {
+    this.name = StringUtils.format(
+        "container_%s_%d",
+        command.getName(),
+        SERVER_ID.incrementAndGet()
+    );
+    this.command = command;
+
+    Integer[] exposedPorts = command.getExposedPorts();
+    servicePort = exposedPorts[0];
+    taskPorts.addAll(Arrays.asList(exposedPorts).subList(1, exposedPorts.length));
+  }
+
+  public DruidContainerResource withPort(int port)
+  {
+    this.servicePort = port;
+    return this;
+  }
+
+  /**
+   * Used to bind the task ports on middle managers.
+   */
+  public DruidContainerResource withTaskPorts(int startPort, int workerCapacity)
+  {
+    taskPorts.clear();
+    for (int i = 0; i < workerCapacity; ++i) {
+      taskPorts.add(startPort + i);
+    }
+    addProperty("druid.worker.capacity", String.valueOf(workerCapacity));
+    addProperty("druid.indexer.runner.startPort", String.valueOf(startPort));
+    addProperty("druid.indexer.runner.endPort", String.valueOf(startPort + workerCapacity));
+
+    return this;
+  }
+
+  public DruidContainerResource withImage(DockerImageName imageName)
+  {
+    this.imageName = imageName;
+    return this;
+  }
+
+  /**
+   * Uses the Docker test image specified by the system property
+   * {@link #PROPERTY_TEST_IMAGE} for this container.
+   */
+  public DruidContainerResource withTestImage()
+  {
+    String imageName = Objects.requireNonNull(
+        System.getProperty(PROPERTY_TEST_IMAGE),
+        StringUtils.format("System property[%s] is not set", PROPERTY_TEST_IMAGE)
+    );
+    return withImage(DockerImageName.parse(imageName));
+  }
+
+  public DruidContainerResource addProperty(String key, String value)
+  {
+    properties.put(key, value);
+    return this;
+  }
+
+  @Override
+  public void beforeStart(EmbeddedDruidCluster cluster)
+  {
+    this.cluster = cluster;
+
+    // Mount directories used by the entire cluster (including embedded servers)
+    this.segmentDeepStorageDirectory = new MountedDir(
+        "/tmp/druid/deep-store",

Review Comment:
   why it needs to be under `tmp` if its inside the container? it could even be `/deep-storage` 



##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/DruidContainerResource.java:
##########
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import org.apache.druid.java.util.common.FileUtils;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.testing.DruidCommand;
+import org.apache.druid.testing.DruidContainer;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.TestcontainerResource;
+import org.testcontainers.containers.BindMode;
+import org.testcontainers.containers.wait.strategy.Wait;
+import org.testcontainers.utility.DockerImageName;
+
+import java.io.File;
+import java.nio.file.Files;
+import java.nio.file.attribute.PosixFilePermissions;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+
+/**
+ * {@link TestcontainerResource} to run Druid services.
+ * Currently, only core extensions can be used out-of-the-box with these containers
+ * such as {@code druid-s3-extensions} or {@code postgresql-metadata-storage},
+ * simply by adding them to {@code druid.extensions.loadList}.
+ * <p>
+ * {@link DruidContainers} should be used only for testing backward compatiblity
+ * or a Docker-specific feature. For all other testing needs, use plain old
+ * {@code EmbeddedDruidServer} as they are much faster, allow easy debugging and
+ * do not require downloading any images.
+ */
+public class DruidContainerResource extends TestcontainerResource<DruidContainer>
+{
+  /**
+   * Java system property to specify the name of the Docker test image.
+   */
+  public static final String PROPERTY_TEST_IMAGE = "druid.testing.docker.image";
+
+  private static final Logger log = new Logger(DruidContainerResource.class);
+
+  /**
+   * Forbidden server properties that may be used by EmbeddedDruidServers but
+   * interfere with the functioning of DruidContainer-based services.
+   */
+  private static final Set<String> FORBIDDEN_PROPERTIES = Set.of(
+      "druid.extensions.modulesForEmbeddedTests",
+      "druid.emitter"
+  );
+
+  /**
+   * A static incremental ID is used instead of a random number to ensure that
+   * tests are more deterministic and easier to debug.
+   */
+  private static final AtomicInteger SERVER_ID = new AtomicInteger(0);
+
+  private final String name;
+  private final DruidCommand command;
+  private final Map<String, String> properties = new HashMap<>();
+  private final List<Integer> taskPorts = new ArrayList<>();
+
+  private int servicePort;
+  private DockerImageName imageName;
+  private EmbeddedDruidCluster cluster;
+
+  private String containerDirectory;
+
+  private MountedDir indexerLogsDeepStorageDirectory;
+  private MountedDir serviceLogsDirectory;
+  private MountedDir segmentDeepStorageDirectory;
+
+  DruidContainerResource(DruidCommand command)
+  {
+    this.name = StringUtils.format(
+        "container_%s_%d",
+        command.getName(),
+        SERVER_ID.incrementAndGet()
+    );
+    this.command = command;
+
+    Integer[] exposedPorts = command.getExposedPorts();
+    servicePort = exposedPorts[0];
+    taskPorts.addAll(Arrays.asList(exposedPorts).subList(1, exposedPorts.length));
+  }
+
+  public DruidContainerResource withPort(int port)
+  {
+    this.servicePort = port;
+    return this;
+  }
+
+  /**
+   * Used to bind the task ports on middle managers.
+   */
+  public DruidContainerResource withTaskPorts(int startPort, int workerCapacity)
+  {
+    taskPorts.clear();
+    for (int i = 0; i < workerCapacity; ++i) {
+      taskPorts.add(startPort + i);
+    }
+    addProperty("druid.worker.capacity", String.valueOf(workerCapacity));
+    addProperty("druid.indexer.runner.startPort", String.valueOf(startPort));
+    addProperty("druid.indexer.runner.endPort", String.valueOf(startPort + workerCapacity));
+
+    return this;
+  }
+
+  public DruidContainerResource withImage(DockerImageName imageName)
+  {
+    this.imageName = imageName;
+    return this;
+  }
+
+  /**
+   * Uses the Docker test image specified by the system property
+   * {@link #PROPERTY_TEST_IMAGE} for this container.
+   */
+  public DruidContainerResource withTestImage()
+  {
+    String imageName = Objects.requireNonNull(
+        System.getProperty(PROPERTY_TEST_IMAGE),
+        StringUtils.format("System property[%s] is not set", PROPERTY_TEST_IMAGE)
+    );
+    return withImage(DockerImageName.parse(imageName));
+  }
+
+  public DruidContainerResource addProperty(String key, String value)
+  {
+    properties.put(key, value);
+    return this;
+  }
+
+  @Override
+  public void beforeStart(EmbeddedDruidCluster cluster)
+  {
+    this.cluster = cluster;
+
+    // Mount directories used by the entire cluster (including embedded servers)
+    this.segmentDeepStorageDirectory = new MountedDir(
+        "/tmp/druid/deep-store",
+        cluster.getTestFolder().getOrCreateFolder("deep-store").getAbsolutePath()
+    );
+    this.indexerLogsDeepStorageDirectory = new MountedDir(
+        "/tmp/druid/indexer-logs",
+        cluster.getTestFolder().getOrCreateFolder("indexer-logs").getAbsolutePath()
+    );
+
+    // Mount directories used by this container for easier debugging with service logs
+    this.containerDirectory = cluster.getTestFolder().getOrCreateFolder(name).getAbsolutePath();
+    this.serviceLogsDirectory = new MountedDir(
+        "/opt/druid/log",
+        containerDirectory + "/log"
+    );
+
+    // Create the log directory upfront to avoid permission issues
+    createLogDirectory(new File(containerDirectory, "log"));
+  }
+
+  @Override
+  protected DruidContainer createContainer()
+  {
+    final List<Integer> exposedPorts = new ArrayList<>(taskPorts);
+    exposedPorts.add(servicePort);
+    log.info(
+        "Starting Druid container[%s] on port[%d] with mounted directory[%s] and exposed ports[%s].",
+        name, servicePort, containerDirectory, exposedPorts
+    );
+
+    final DruidContainer container = new DruidContainer(command, imageName)
+        .withExposedPorts(exposedPorts.toArray(new Integer[0]))
+        .withCommonProperties(getCommonProperties())
+        .withServiceProperties(getServerProperties())
+        .withFileSystemBind(
+            segmentDeepStorageDirectory.hostPath,
+            segmentDeepStorageDirectory.containerPath,
+            BindMode.READ_WRITE
+        )
+        .withFileSystemBind(
+            indexerLogsDeepStorageDirectory.hostPath,
+            indexerLogsDeepStorageDirectory.containerPath,
+            BindMode.READ_WRITE
+        )
+        .withFileSystemBind(
+            serviceLogsDirectory.hostPath,
+            serviceLogsDirectory.containerPath,
+            BindMode.READ_WRITE
+        )
+        .withEnv(
+            Map.of(
+                "DRUID_SET_HOST_IP", "0",
+                "DRUID_SET_HOST", "0"
+            )
+        )
+        .waitingFor(Wait.forHttp("/status/health").forPort(servicePort));
+
+    // Bind the ports statically (rather than using a mapped port) to the same
+    // value used in `druid.plaintextPort` to make this node discoverable
+    // by other services (both embedded and dockerized).
+    List<String> portBindings = exposedPorts.stream().map(
+        port -> StringUtils.format("%d:%d", port, port)
+    ).collect(Collectors.toList());
+    container.setPortBindings(portBindings);

Review Comment:
   there could be an `exposePorts` which accepts the list...so no `map` and other stuff is at the place its being used



##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/DruidContainerResource.java:
##########
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import org.apache.druid.java.util.common.FileUtils;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.testing.DruidCommand;
+import org.apache.druid.testing.DruidContainer;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.TestcontainerResource;
+import org.testcontainers.containers.BindMode;
+import org.testcontainers.containers.wait.strategy.Wait;
+import org.testcontainers.utility.DockerImageName;
+
+import java.io.File;
+import java.nio.file.Files;
+import java.nio.file.attribute.PosixFilePermissions;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+
+/**
+ * {@link TestcontainerResource} to run Druid services.
+ * Currently, only core extensions can be used out-of-the-box with these containers
+ * such as {@code druid-s3-extensions} or {@code postgresql-metadata-storage},
+ * simply by adding them to {@code druid.extensions.loadList}.
+ * <p>
+ * {@link DruidContainers} should be used only for testing backward compatiblity
+ * or a Docker-specific feature. For all other testing needs, use plain old
+ * {@code EmbeddedDruidServer} as they are much faster, allow easy debugging and
+ * do not require downloading any images.
+ */
+public class DruidContainerResource extends TestcontainerResource<DruidContainer>
+{
+  /**
+   * Java system property to specify the name of the Docker test image.
+   */
+  public static final String PROPERTY_TEST_IMAGE = "druid.testing.docker.image";
+
+  private static final Logger log = new Logger(DruidContainerResource.class);
+
+  /**
+   * Forbidden server properties that may be used by EmbeddedDruidServers but
+   * interfere with the functioning of DruidContainer-based services.
+   */
+  private static final Set<String> FORBIDDEN_PROPERTIES = Set.of(
+      "druid.extensions.modulesForEmbeddedTests",
+      "druid.emitter"
+  );
+
+  /**
+   * A static incremental ID is used instead of a random number to ensure that
+   * tests are more deterministic and easier to debug.
+   */
+  private static final AtomicInteger SERVER_ID = new AtomicInteger(0);
+
+  private final String name;
+  private final DruidCommand command;
+  private final Map<String, String> properties = new HashMap<>();
+  private final List<Integer> taskPorts = new ArrayList<>();
+
+  private int servicePort;
+  private DockerImageName imageName;
+  private EmbeddedDruidCluster cluster;
+
+  private String containerDirectory;
+
+  private MountedDir indexerLogsDeepStorageDirectory;
+  private MountedDir serviceLogsDirectory;
+  private MountedDir segmentDeepStorageDirectory;
+
+  DruidContainerResource(DruidCommand command)
+  {
+    this.name = StringUtils.format(
+        "container_%s_%d",
+        command.getName(),
+        SERVER_ID.incrementAndGet()
+    );
+    this.command = command;
+
+    Integer[] exposedPorts = command.getExposedPorts();
+    servicePort = exposedPorts[0];
+    taskPorts.addAll(Arrays.asList(exposedPorts).subList(1, exposedPorts.length));
+  }
+
+  public DruidContainerResource withPort(int port)
+  {
+    this.servicePort = port;
+    return this;
+  }
+
+  /**
+   * Used to bind the task ports on middle managers.
+   */
+  public DruidContainerResource withTaskPorts(int startPort, int workerCapacity)
+  {
+    taskPorts.clear();
+    for (int i = 0; i < workerCapacity; ++i) {
+      taskPorts.add(startPort + i);
+    }
+    addProperty("druid.worker.capacity", String.valueOf(workerCapacity));
+    addProperty("druid.indexer.runner.startPort", String.valueOf(startPort));
+    addProperty("druid.indexer.runner.endPort", String.valueOf(startPort + workerCapacity));
+
+    return this;
+  }
+
+  public DruidContainerResource withImage(DockerImageName imageName)
+  {
+    this.imageName = imageName;
+    return this;
+  }
+
+  /**
+   * Uses the Docker test image specified by the system property
+   * {@link #PROPERTY_TEST_IMAGE} for this container.
+   */
+  public DruidContainerResource withTestImage()
+  {
+    String imageName = Objects.requireNonNull(
+        System.getProperty(PROPERTY_TEST_IMAGE),
+        StringUtils.format("System property[%s] is not set", PROPERTY_TEST_IMAGE)
+    );
+    return withImage(DockerImageName.parse(imageName));
+  }
+
+  public DruidContainerResource addProperty(String key, String value)
+  {
+    properties.put(key, value);
+    return this;
+  }
+
+  @Override
+  public void beforeStart(EmbeddedDruidCluster cluster)
+  {
+    this.cluster = cluster;
+
+    // Mount directories used by the entire cluster (including embedded servers)
+    this.segmentDeepStorageDirectory = new MountedDir(
+        "/tmp/druid/deep-store",
+        cluster.getTestFolder().getOrCreateFolder("deep-store").getAbsolutePath()
+    );
+    this.indexerLogsDeepStorageDirectory = new MountedDir(
+        "/tmp/druid/indexer-logs",
+        cluster.getTestFolder().getOrCreateFolder("indexer-logs").getAbsolutePath()
+    );
+
+    // Mount directories used by this container for easier debugging with service logs
+    this.containerDirectory = cluster.getTestFolder().getOrCreateFolder(name).getAbsolutePath();
+    this.serviceLogsDirectory = new MountedDir(
+        "/opt/druid/log",
+        containerDirectory + "/log"
+    );
+
+    // Create the log directory upfront to avoid permission issues
+    createLogDirectory(new File(containerDirectory, "log"));
+  }
+
+  @Override
+  protected DruidContainer createContainer()
+  {
+    final List<Integer> exposedPorts = new ArrayList<>(taskPorts);
+    exposedPorts.add(servicePort);
+    log.info(
+        "Starting Druid container[%s] on port[%d] with mounted directory[%s] and exposed ports[%s].",
+        name, servicePort, containerDirectory, exposedPorts
+    );
+
+    final DruidContainer container = new DruidContainer(command, imageName)
+        .withExposedPorts(exposedPorts.toArray(new Integer[0]))
+        .withCommonProperties(getCommonProperties())
+        .withServiceProperties(getServerProperties())
+        .withFileSystemBind(
+            segmentDeepStorageDirectory.hostPath,
+            segmentDeepStorageDirectory.containerPath,
+            BindMode.READ_WRITE
+        )
+        .withFileSystemBind(
+            indexerLogsDeepStorageDirectory.hostPath,
+            indexerLogsDeepStorageDirectory.containerPath,
+            BindMode.READ_WRITE
+        )
+        .withFileSystemBind(
+            serviceLogsDirectory.hostPath,
+            serviceLogsDirectory.containerPath,
+            BindMode.READ_WRITE
+        )
+        .withEnv(
+            Map.of(
+                "DRUID_SET_HOST_IP", "0",
+                "DRUID_SET_HOST", "0"
+            )
+        )
+        .waitingFor(Wait.forHttp("/status/health").forPort(servicePort));
+
+    // Bind the ports statically (rather than using a mapped port) to the same
+    // value used in `druid.plaintextPort` to make this node discoverable
+    // by other services (both embedded and dockerized).
+    List<String> portBindings = exposedPorts.stream().map(
+        port -> StringUtils.format("%d:%d", port, port)
+    ).collect(Collectors.toList());
+    container.setPortBindings(portBindings);
+
+    return container;
+  }
+
+  /**
+   * Writes the common properties to a file.
+   *
+   * @return Name of the file.
+   */
+  private Properties getCommonProperties()
+  {
+    final Properties commonProperties = new Properties();
+    commonProperties.putAll(cluster.getCommonProperties());
+    FORBIDDEN_PROPERTIES.forEach(commonProperties::remove);
+
+    commonProperties.setProperty(
+        "druid.storage.storageDirectory",
+        segmentDeepStorageDirectory.containerPath
+    );
+    commonProperties.setProperty(
+        "druid.indexer.logs.directory",
+        indexerLogsDeepStorageDirectory.containerPath
+    );
+
+    log.info(
+        "Writing common properties for Druid container[%s]: [%s]",
+        name, commonProperties
+    );
+
+    return commonProperties;
+  }
+
+  /**
+   * Writes the server properties to a file.
+   *
+   * @return Name of the file.
+   */
+  private Properties getServerProperties()
+  {
+    FORBIDDEN_PROPERTIES.forEach(properties::remove);
+    addProperty("druid.host", EmbeddedDruidCluster.getDefaultHost());
+    addProperty("druid.plaintextPort", String.valueOf(servicePort));
+
+    final Properties serverProperties = new Properties();
+    serverProperties.putAll(properties);
+
+    log.info(
+        "Writing runtime properties for Druid container[%s]: [%s]",
+        name, serverProperties
+    );
+
+    return serverProperties;
+  }
+
+  private static void createLogDirectory(File dir)
+  {
+    try {
+      FileUtils.mkdirp(dir);
+      Files.setPosixFilePermissions(dir.toPath(), PosixFilePermissions.fromString("rwxrwxrwx"));
+    }
+    catch (Exception e) {
+      throw new RuntimeException(e);
+    }
+  }
+
+  @Override
+  public String toString()
+  {
+    return name;
+  }
+
+  private static class MountedDir
+  {
+    final String hostPath;
+    final String containerPath;

Review Comment:
   these seems like paths - throwing away `File` and `Path` will make it harder to work with them....



##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/DruidContainerResource.java:
##########
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import org.apache.druid.java.util.common.FileUtils;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.testing.DruidCommand;
+import org.apache.druid.testing.DruidContainer;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.TestcontainerResource;
+import org.testcontainers.containers.BindMode;
+import org.testcontainers.containers.wait.strategy.Wait;
+import org.testcontainers.utility.DockerImageName;
+
+import java.io.File;
+import java.nio.file.Files;
+import java.nio.file.attribute.PosixFilePermissions;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+
+/**
+ * {@link TestcontainerResource} to run Druid services.
+ * Currently, only core extensions can be used out-of-the-box with these containers
+ * such as {@code druid-s3-extensions} or {@code postgresql-metadata-storage},
+ * simply by adding them to {@code druid.extensions.loadList}.
+ * <p>
+ * {@link DruidContainers} should be used only for testing backward compatiblity
+ * or a Docker-specific feature. For all other testing needs, use plain old
+ * {@code EmbeddedDruidServer} as they are much faster, allow easy debugging and
+ * do not require downloading any images.
+ */
+public class DruidContainerResource extends TestcontainerResource<DruidContainer>
+{
+  /**
+   * Java system property to specify the name of the Docker test image.
+   */
+  public static final String PROPERTY_TEST_IMAGE = "druid.testing.docker.image";
+
+  private static final Logger log = new Logger(DruidContainerResource.class);
+
+  /**
+   * Forbidden server properties that may be used by EmbeddedDruidServers but
+   * interfere with the functioning of DruidContainer-based services.
+   */
+  private static final Set<String> FORBIDDEN_PROPERTIES = Set.of(
+      "druid.extensions.modulesForEmbeddedTests",
+      "druid.emitter"
+  );
+
+  /**
+   * A static incremental ID is used instead of a random number to ensure that
+   * tests are more deterministic and easier to debug.
+   */
+  private static final AtomicInteger SERVER_ID = new AtomicInteger(0);
+
+  private final String name;
+  private final DruidCommand command;
+  private final Map<String, String> properties = new HashMap<>();
+  private final List<Integer> taskPorts = new ArrayList<>();
+
+  private int servicePort;
+  private DockerImageName imageName;
+  private EmbeddedDruidCluster cluster;
+
+  private String containerDirectory;
+
+  private MountedDir indexerLogsDeepStorageDirectory;
+  private MountedDir serviceLogsDirectory;
+  private MountedDir segmentDeepStorageDirectory;
+
+  DruidContainerResource(DruidCommand command)
+  {
+    this.name = StringUtils.format(
+        "container_%s_%d",
+        command.getName(),
+        SERVER_ID.incrementAndGet()
+    );
+    this.command = command;
+
+    Integer[] exposedPorts = command.getExposedPorts();
+    servicePort = exposedPorts[0];
+    taskPorts.addAll(Arrays.asList(exposedPorts).subList(1, exposedPorts.length));
+  }
+
+  public DruidContainerResource withPort(int port)
+  {
+    this.servicePort = port;
+    return this;
+  }
+
+  /**
+   * Used to bind the task ports on middle managers.
+   */
+  public DruidContainerResource withTaskPorts(int startPort, int workerCapacity)
+  {
+    taskPorts.clear();
+    for (int i = 0; i < workerCapacity; ++i) {
+      taskPorts.add(startPort + i);
+    }
+    addProperty("druid.worker.capacity", String.valueOf(workerCapacity));
+    addProperty("druid.indexer.runner.startPort", String.valueOf(startPort));
+    addProperty("druid.indexer.runner.endPort", String.valueOf(startPort + workerCapacity));
+
+    return this;
+  }
+
+  public DruidContainerResource withImage(DockerImageName imageName)
+  {
+    this.imageName = imageName;
+    return this;
+  }
+
+  /**
+   * Uses the Docker test image specified by the system property
+   * {@link #PROPERTY_TEST_IMAGE} for this container.
+   */
+  public DruidContainerResource withTestImage()
+  {
+    String imageName = Objects.requireNonNull(
+        System.getProperty(PROPERTY_TEST_IMAGE),
+        StringUtils.format("System property[%s] is not set", PROPERTY_TEST_IMAGE)
+    );
+    return withImage(DockerImageName.parse(imageName));
+  }
+
+  public DruidContainerResource addProperty(String key, String value)
+  {
+    properties.put(key, value);
+    return this;
+  }
+
+  @Override
+  public void beforeStart(EmbeddedDruidCluster cluster)
+  {
+    this.cluster = cluster;
+
+    // Mount directories used by the entire cluster (including embedded servers)
+    this.segmentDeepStorageDirectory = new MountedDir(
+        "/tmp/druid/deep-store",
+        cluster.getTestFolder().getOrCreateFolder("deep-store").getAbsolutePath()
+    );
+    this.indexerLogsDeepStorageDirectory = new MountedDir(
+        "/tmp/druid/indexer-logs",
+        cluster.getTestFolder().getOrCreateFolder("indexer-logs").getAbsolutePath()
+    );
+
+    // Mount directories used by this container for easier debugging with service logs
+    this.containerDirectory = cluster.getTestFolder().getOrCreateFolder(name).getAbsolutePath();
+    this.serviceLogsDirectory = new MountedDir(
+        "/opt/druid/log",
+        containerDirectory + "/log"
+    );
+
+    // Create the log directory upfront to avoid permission issues
+    createLogDirectory(new File(containerDirectory, "log"));
+  }
+
+  @Override
+  protected DruidContainer createContainer()
+  {
+    final List<Integer> exposedPorts = new ArrayList<>(taskPorts);
+    exposedPorts.add(servicePort);
+    log.info(
+        "Starting Druid container[%s] on port[%d] with mounted directory[%s] and exposed ports[%s].",
+        name, servicePort, containerDirectory, exposedPorts
+    );
+
+    final DruidContainer container = new DruidContainer(command, imageName)
+        .withExposedPorts(exposedPorts.toArray(new Integer[0]))

Review Comment:
   there could be a method which accepts `List` ?



##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/DruidContainers.java:
##########
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import org.apache.druid.testing.DruidCommand;
+
+/**
+ * Factory for {@link DruidContainerResource} that can run specific services.
+ *
+ * @see #newOverlord()
+ * @see #newCoordinator()
+ */
+public final class DruidContainers
+{
+  private DruidContainers()
+  {
+    // no instantiation
+  }
+
+  /**
+   * Creates a new {@link DruidContainerResource} to run a Coordinator node.
+   */
+  public static DruidContainerResource newCoordinator()
+  {
+    return new DruidContainerResource(DruidCommand.COORDINATOR)
+        .addProperty("druid.coordinator.startDelay", "PT0.1S")
+        .addProperty("druid.coordinator.period", "PT0.5S")
+        .addProperty("druid.manager.segments.pollDuration", "PT0.1S");
+  }
+
+  /**
+   * Creates a new {@link DruidContainerResource} to run an Overlord node.
+   */
+  public static DruidContainerResource newOverlord()
+  {
+    // Keep a small sync timeout so that Peons and Indexers are not stuck
+    // handling a change request when Overlord has already shutdown
+    return new DruidContainerResource(DruidCommand.OVERLORD)
+        .addProperty("druid.indexer.storage.type", "metadata")
+        .addProperty("druid.indexer.queue.startDelay", "PT0S")
+        .addProperty("druid.indexer.queue.restartDelay", "PT0S")
+        .addProperty("druid.indexer.runner.syncRequestTimeout", "PT1S");
+  }
+
+  /**
+   * Creates a new {@link DruidContainerResource} to run an Indexer node.
+   */
+  public static DruidContainerResource newIndexer()
+  {
+    return new DruidContainerResource(DruidCommand.INDEXER)
+        .addProperty("druid.lookup.enableLookupSyncOnStartup", "false")
+        .addProperty("druid.processing.buffer.sizeBytes", "50MiB")
+        .addProperty("druid.processing.numMergeBuffers", "2")
+        .addProperty("druid.processing.numThreads", "5");
+  }

Review Comment:
   feels like the `DruidCommand`  was over and underused at the same time...
   * it contains some stuff so `new DruidContainerResource(DruidCommand.INDEXER)` could create a valid object
   * its underused as a good one will have these 4 properties set...
   
   feels like there should be a `DruidOverlordContainer` / `DruidIndexerContainer` and similar....it could still be an innerclass here... 
   



##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/DruidContainerResource.java:
##########
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import org.apache.druid.java.util.common.FileUtils;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.testing.DruidCommand;
+import org.apache.druid.testing.DruidContainer;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.TestcontainerResource;
+import org.testcontainers.containers.BindMode;
+import org.testcontainers.containers.wait.strategy.Wait;
+import org.testcontainers.utility.DockerImageName;
+
+import java.io.File;
+import java.nio.file.Files;
+import java.nio.file.attribute.PosixFilePermissions;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+
+/**
+ * {@link TestcontainerResource} to run Druid services.
+ * Currently, only core extensions can be used out-of-the-box with these containers
+ * such as {@code druid-s3-extensions} or {@code postgresql-metadata-storage},
+ * simply by adding them to {@code druid.extensions.loadList}.
+ * <p>
+ * {@link DruidContainers} should be used only for testing backward compatiblity
+ * or a Docker-specific feature. For all other testing needs, use plain old
+ * {@code EmbeddedDruidServer} as they are much faster, allow easy debugging and
+ * do not require downloading any images.
+ */
+public class DruidContainerResource extends TestcontainerResource<DruidContainer>
+{
+  /**
+   * Java system property to specify the name of the Docker test image.
+   */
+  public static final String PROPERTY_TEST_IMAGE = "druid.testing.docker.image";
+
+  private static final Logger log = new Logger(DruidContainerResource.class);
+
+  /**
+   * Forbidden server properties that may be used by EmbeddedDruidServers but
+   * interfere with the functioning of DruidContainer-based services.
+   */
+  private static final Set<String> FORBIDDEN_PROPERTIES = Set.of(
+      "druid.extensions.modulesForEmbeddedTests",
+      "druid.emitter"
+  );
+
+  /**
+   * A static incremental ID is used instead of a random number to ensure that
+   * tests are more deterministic and easier to debug.
+   */
+  private static final AtomicInteger SERVER_ID = new AtomicInteger(0);
+
+  private final String name;
+  private final DruidCommand command;
+  private final Map<String, String> properties = new HashMap<>();
+  private final List<Integer> taskPorts = new ArrayList<>();
+
+  private int servicePort;
+  private DockerImageName imageName;
+  private EmbeddedDruidCluster cluster;
+
+  private String containerDirectory;
+
+  private MountedDir indexerLogsDeepStorageDirectory;
+  private MountedDir serviceLogsDirectory;
+  private MountedDir segmentDeepStorageDirectory;
+
+  DruidContainerResource(DruidCommand command)
+  {
+    this.name = StringUtils.format(
+        "container_%s_%d",
+        command.getName(),
+        SERVER_ID.incrementAndGet()
+    );
+    this.command = command;
+
+    Integer[] exposedPorts = command.getExposedPorts();
+    servicePort = exposedPorts[0];
+    taskPorts.addAll(Arrays.asList(exposedPorts).subList(1, exposedPorts.length));
+  }
+
+  public DruidContainerResource withPort(int port)
+  {
+    this.servicePort = port;
+    return this;
+  }
+
+  /**
+   * Used to bind the task ports on middle managers.
+   */
+  public DruidContainerResource withTaskPorts(int startPort, int workerCapacity)
+  {
+    taskPorts.clear();
+    for (int i = 0; i < workerCapacity; ++i) {
+      taskPorts.add(startPort + i);
+    }
+    addProperty("druid.worker.capacity", String.valueOf(workerCapacity));
+    addProperty("druid.indexer.runner.startPort", String.valueOf(startPort));
+    addProperty("druid.indexer.runner.endPort", String.valueOf(startPort + workerCapacity));
+
+    return this;
+  }
+
+  public DruidContainerResource withImage(DockerImageName imageName)
+  {
+    this.imageName = imageName;
+    return this;
+  }
+
+  /**
+   * Uses the Docker test image specified by the system property
+   * {@link #PROPERTY_TEST_IMAGE} for this container.
+   */
+  public DruidContainerResource withTestImage()
+  {
+    String imageName = Objects.requireNonNull(
+        System.getProperty(PROPERTY_TEST_IMAGE),
+        StringUtils.format("System property[%s] is not set", PROPERTY_TEST_IMAGE)
+    );
+    return withImage(DockerImageName.parse(imageName));
+  }
+
+  public DruidContainerResource addProperty(String key, String value)
+  {
+    properties.put(key, value);
+    return this;
+  }
+
+  @Override
+  public void beforeStart(EmbeddedDruidCluster cluster)
+  {
+    this.cluster = cluster;
+
+    // Mount directories used by the entire cluster (including embedded servers)
+    this.segmentDeepStorageDirectory = new MountedDir(
+        "/tmp/druid/deep-store",
+        cluster.getTestFolder().getOrCreateFolder("deep-store").getAbsolutePath()
+    );
+    this.indexerLogsDeepStorageDirectory = new MountedDir(
+        "/tmp/druid/indexer-logs",
+        cluster.getTestFolder().getOrCreateFolder("indexer-logs").getAbsolutePath()
+    );
+
+    // Mount directories used by this container for easier debugging with service logs
+    this.containerDirectory = cluster.getTestFolder().getOrCreateFolder(name).getAbsolutePath();
+    this.serviceLogsDirectory = new MountedDir(
+        "/opt/druid/log",

Review Comment:
   not sure about this folder layout....at first time I would never look at `/opt/` for logs...



##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/DruidContainerResource.java:
##########
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import org.apache.druid.java.util.common.FileUtils;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.testing.DruidCommand;
+import org.apache.druid.testing.DruidContainer;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.TestcontainerResource;
+import org.testcontainers.containers.BindMode;
+import org.testcontainers.containers.wait.strategy.Wait;
+import org.testcontainers.utility.DockerImageName;
+
+import java.io.File;
+import java.nio.file.Files;
+import java.nio.file.attribute.PosixFilePermissions;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+
+/**
+ * {@link TestcontainerResource} to run Druid services.
+ * Currently, only core extensions can be used out-of-the-box with these containers
+ * such as {@code druid-s3-extensions} or {@code postgresql-metadata-storage},
+ * simply by adding them to {@code druid.extensions.loadList}.
+ * <p>
+ * {@link DruidContainers} should be used only for testing backward compatiblity
+ * or a Docker-specific feature. For all other testing needs, use plain old
+ * {@code EmbeddedDruidServer} as they are much faster, allow easy debugging and
+ * do not require downloading any images.
+ */
+public class DruidContainerResource extends TestcontainerResource<DruidContainer>
+{
+  /**
+   * Java system property to specify the name of the Docker test image.
+   */
+  public static final String PROPERTY_TEST_IMAGE = "druid.testing.docker.image";
+
+  private static final Logger log = new Logger(DruidContainerResource.class);
+
+  /**
+   * Forbidden server properties that may be used by EmbeddedDruidServers but
+   * interfere with the functioning of DruidContainer-based services.
+   */
+  private static final Set<String> FORBIDDEN_PROPERTIES = Set.of(
+      "druid.extensions.modulesForEmbeddedTests",
+      "druid.emitter"
+  );
+
+  /**
+   * A static incremental ID is used instead of a random number to ensure that
+   * tests are more deterministic and easier to debug.
+   */
+  private static final AtomicInteger SERVER_ID = new AtomicInteger(0);
+
+  private final String name;
+  private final DruidCommand command;
+  private final Map<String, String> properties = new HashMap<>();
+  private final List<Integer> taskPorts = new ArrayList<>();
+
+  private int servicePort;
+  private DockerImageName imageName;
+  private EmbeddedDruidCluster cluster;
+
+  private String containerDirectory;
+
+  private MountedDir indexerLogsDeepStorageDirectory;
+  private MountedDir serviceLogsDirectory;
+  private MountedDir segmentDeepStorageDirectory;
+
+  DruidContainerResource(DruidCommand command)
+  {
+    this.name = StringUtils.format(
+        "container_%s_%d",
+        command.getName(),
+        SERVER_ID.incrementAndGet()
+    );
+    this.command = command;
+
+    Integer[] exposedPorts = command.getExposedPorts();
+    servicePort = exposedPorts[0];
+    taskPorts.addAll(Arrays.asList(exposedPorts).subList(1, exposedPorts.length));
+  }
+
+  public DruidContainerResource withPort(int port)
+  {
+    this.servicePort = port;
+    return this;
+  }
+
+  /**
+   * Used to bind the task ports on middle managers.
+   */
+  public DruidContainerResource withTaskPorts(int startPort, int workerCapacity)
+  {
+    taskPorts.clear();
+    for (int i = 0; i < workerCapacity; ++i) {
+      taskPorts.add(startPort + i);
+    }
+    addProperty("druid.worker.capacity", String.valueOf(workerCapacity));
+    addProperty("druid.indexer.runner.startPort", String.valueOf(startPort));
+    addProperty("druid.indexer.runner.endPort", String.valueOf(startPort + workerCapacity));
+
+    return this;
+  }
+
+  public DruidContainerResource withImage(DockerImageName imageName)
+  {
+    this.imageName = imageName;
+    return this;
+  }
+
+  /**
+   * Uses the Docker test image specified by the system property
+   * {@link #PROPERTY_TEST_IMAGE} for this container.
+   */
+  public DruidContainerResource withTestImage()
+  {
+    String imageName = Objects.requireNonNull(
+        System.getProperty(PROPERTY_TEST_IMAGE),
+        StringUtils.format("System property[%s] is not set", PROPERTY_TEST_IMAGE)
+    );
+    return withImage(DockerImageName.parse(imageName));
+  }
+
+  public DruidContainerResource addProperty(String key, String value)
+  {
+    properties.put(key, value);
+    return this;
+  }
+
+  @Override
+  public void beforeStart(EmbeddedDruidCluster cluster)
+  {
+    this.cluster = cluster;
+
+    // Mount directories used by the entire cluster (including embedded servers)
+    this.segmentDeepStorageDirectory = new MountedDir(
+        "/tmp/druid/deep-store",
+        cluster.getTestFolder().getOrCreateFolder("deep-store").getAbsolutePath()
+    );
+    this.indexerLogsDeepStorageDirectory = new MountedDir(
+        "/tmp/druid/indexer-logs",
+        cluster.getTestFolder().getOrCreateFolder("indexer-logs").getAbsolutePath()
+    );
+
+    // Mount directories used by this container for easier debugging with service logs
+    this.containerDirectory = cluster.getTestFolder().getOrCreateFolder(name).getAbsolutePath();
+    this.serviceLogsDirectory = new MountedDir(
+        "/opt/druid/log",
+        containerDirectory + "/log"
+    );
+
+    // Create the log directory upfront to avoid permission issues
+    createLogDirectory(new File(containerDirectory, "log"));
+  }
+
+  @Override
+  protected DruidContainer createContainer()
+  {
+    final List<Integer> exposedPorts = new ArrayList<>(taskPorts);
+    exposedPorts.add(servicePort);
+    log.info(
+        "Starting Druid container[%s] on port[%d] with mounted directory[%s] and exposed ports[%s].",
+        name, servicePort, containerDirectory, exposedPorts
+    );
+
+    final DruidContainer container = new DruidContainer(command, imageName)
+        .withExposedPorts(exposedPorts.toArray(new Integer[0]))
+        .withCommonProperties(getCommonProperties())
+        .withServiceProperties(getServerProperties())
+        .withFileSystemBind(
+            segmentDeepStorageDirectory.hostPath,
+            segmentDeepStorageDirectory.containerPath,
+            BindMode.READ_WRITE
+        )
+        .withFileSystemBind(
+            indexerLogsDeepStorageDirectory.hostPath,
+            indexerLogsDeepStorageDirectory.containerPath,
+            BindMode.READ_WRITE
+        )
+        .withFileSystemBind(
+            serviceLogsDirectory.hostPath,
+            serviceLogsDirectory.containerPath,
+            BindMode.READ_WRITE
+        )
+        .withEnv(
+            Map.of(
+                "DRUID_SET_HOST_IP", "0",
+                "DRUID_SET_HOST", "0"
+            )
+        )
+        .waitingFor(Wait.forHttp("/status/health").forPort(servicePort));
+
+    // Bind the ports statically (rather than using a mapped port) to the same
+    // value used in `druid.plaintextPort` to make this node discoverable
+    // by other services (both embedded and dockerized).
+    List<String> portBindings = exposedPorts.stream().map(
+        port -> StringUtils.format("%d:%d", port, port)
+    ).collect(Collectors.toList());
+    container.setPortBindings(portBindings);
+
+    return container;
+  }
+
+  /**
+   * Writes the common properties to a file.
+   *
+   * @return Name of the file.
+   */
+  private Properties getCommonProperties()
+  {
+    final Properties commonProperties = new Properties();
+    commonProperties.putAll(cluster.getCommonProperties());
+    FORBIDDEN_PROPERTIES.forEach(commonProperties::remove);
+
+    commonProperties.setProperty(
+        "druid.storage.storageDirectory",
+        segmentDeepStorageDirectory.containerPath
+    );
+    commonProperties.setProperty(
+        "druid.indexer.logs.directory",
+        indexerLogsDeepStorageDirectory.containerPath
+    );
+
+    log.info(
+        "Writing common properties for Druid container[%s]: [%s]",
+        name, commonProperties
+    );
+
+    return commonProperties;
+  }
+
+  /**
+   * Writes the server properties to a file.
+   *
+   * @return Name of the file.
+   */

Review Comment:
   the apidoc doesn't match the function's mission; and its also feels like its irrelevant



##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/DruidContainers.java:
##########
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import org.apache.druid.testing.DruidCommand;
+
+/**
+ * Factory for {@link DruidContainerResource} that can run specific services.
+ *
+ * @see #newOverlord()
+ * @see #newCoordinator()
+ */
+public final class DruidContainers
+{
+  private DruidContainers()
+  {
+    // no instantiation
+  }
+
+  /**
+   * Creates a new {@link DruidContainerResource} to run a Coordinator node.
+   */

Review Comment:
   nit: this apidoc doesn't add any value beyond the function signature:
   ```
   public static DruidContainerResource newCoordinator()
   ```
   
   this is true for a lot of places....I think adding apidoc like that could be harmfull as they will become stale after some time.
   



##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/DruidContainerResource.java:
##########
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import org.apache.druid.java.util.common.FileUtils;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.testing.DruidCommand;
+import org.apache.druid.testing.DruidContainer;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.TestcontainerResource;
+import org.testcontainers.containers.BindMode;
+import org.testcontainers.containers.wait.strategy.Wait;
+import org.testcontainers.utility.DockerImageName;
+
+import java.io.File;
+import java.nio.file.Files;
+import java.nio.file.attribute.PosixFilePermissions;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+
+/**
+ * {@link TestcontainerResource} to run Druid services.
+ * Currently, only core extensions can be used out-of-the-box with these containers
+ * such as {@code druid-s3-extensions} or {@code postgresql-metadata-storage},
+ * simply by adding them to {@code druid.extensions.loadList}.
+ * <p>
+ * {@link DruidContainers} should be used only for testing backward compatiblity
+ * or a Docker-specific feature. For all other testing needs, use plain old
+ * {@code EmbeddedDruidServer} as they are much faster, allow easy debugging and
+ * do not require downloading any images.
+ */
+public class DruidContainerResource extends TestcontainerResource<DruidContainer>
+{
+  /**
+   * Java system property to specify the name of the Docker test image.
+   */
+  public static final String PROPERTY_TEST_IMAGE = "druid.testing.docker.image";
+
+  private static final Logger log = new Logger(DruidContainerResource.class);
+
+  /**
+   * Forbidden server properties that may be used by EmbeddedDruidServers but
+   * interfere with the functioning of DruidContainer-based services.
+   */
+  private static final Set<String> FORBIDDEN_PROPERTIES = Set.of(
+      "druid.extensions.modulesForEmbeddedTests",
+      "druid.emitter"
+  );
+
+  /**
+   * A static incremental ID is used instead of a random number to ensure that
+   * tests are more deterministic and easier to debug.
+   */
+  private static final AtomicInteger SERVER_ID = new AtomicInteger(0);
+
+  private final String name;
+  private final DruidCommand command;
+  private final Map<String, String> properties = new HashMap<>();
+  private final List<Integer> taskPorts = new ArrayList<>();
+
+  private int servicePort;
+  private DockerImageName imageName;
+  private EmbeddedDruidCluster cluster;
+
+  private String containerDirectory;
+
+  private MountedDir indexerLogsDeepStorageDirectory;
+  private MountedDir serviceLogsDirectory;
+  private MountedDir segmentDeepStorageDirectory;
+
+  DruidContainerResource(DruidCommand command)
+  {
+    this.name = StringUtils.format(
+        "container_%s_%d",
+        command.getName(),
+        SERVER_ID.incrementAndGet()
+    );
+    this.command = command;
+
+    Integer[] exposedPorts = command.getExposedPorts();
+    servicePort = exposedPorts[0];
+    taskPorts.addAll(Arrays.asList(exposedPorts).subList(1, exposedPorts.length));
+  }
+
+  public DruidContainerResource withPort(int port)
+  {
+    this.servicePort = port;
+    return this;
+  }
+
+  /**
+   * Used to bind the task ports on middle managers.
+   */
+  public DruidContainerResource withTaskPorts(int startPort, int workerCapacity)
+  {
+    taskPorts.clear();
+    for (int i = 0; i < workerCapacity; ++i) {
+      taskPorts.add(startPort + i);
+    }
+    addProperty("druid.worker.capacity", String.valueOf(workerCapacity));
+    addProperty("druid.indexer.runner.startPort", String.valueOf(startPort));
+    addProperty("druid.indexer.runner.endPort", String.valueOf(startPort + workerCapacity));
+
+    return this;
+  }
+
+  public DruidContainerResource withImage(DockerImageName imageName)
+  {
+    this.imageName = imageName;
+    return this;
+  }
+
+  /**
+   * Uses the Docker test image specified by the system property
+   * {@link #PROPERTY_TEST_IMAGE} for this container.
+   */
+  public DruidContainerResource withTestImage()
+  {
+    String imageName = Objects.requireNonNull(
+        System.getProperty(PROPERTY_TEST_IMAGE),
+        StringUtils.format("System property[%s] is not set", PROPERTY_TEST_IMAGE)
+    );
+    return withImage(DockerImageName.parse(imageName));
+  }
+
+  public DruidContainerResource addProperty(String key, String value)
+  {
+    properties.put(key, value);
+    return this;
+  }
+
+  @Override
+  public void beforeStart(EmbeddedDruidCluster cluster)
+  {
+    this.cluster = cluster;
+
+    // Mount directories used by the entire cluster (including embedded servers)
+    this.segmentDeepStorageDirectory = new MountedDir(
+        "/tmp/druid/deep-store",
+        cluster.getTestFolder().getOrCreateFolder("deep-store").getAbsolutePath()
+    );
+    this.indexerLogsDeepStorageDirectory = new MountedDir(
+        "/tmp/druid/indexer-logs",
+        cluster.getTestFolder().getOrCreateFolder("indexer-logs").getAbsolutePath()
+    );
+
+    // Mount directories used by this container for easier debugging with service logs
+    this.containerDirectory = cluster.getTestFolder().getOrCreateFolder(name).getAbsolutePath();
+    this.serviceLogsDirectory = new MountedDir(
+        "/opt/druid/log",
+        containerDirectory + "/log"
+    );
+
+    // Create the log directory upfront to avoid permission issues
+    createLogDirectory(new File(containerDirectory, "log"));
+  }
+
+  @Override
+  protected DruidContainer createContainer()
+  {
+    final List<Integer> exposedPorts = new ArrayList<>(taskPorts);
+    exposedPorts.add(servicePort);
+    log.info(
+        "Starting Druid container[%s] on port[%d] with mounted directory[%s] and exposed ports[%s].",
+        name, servicePort, containerDirectory, exposedPorts
+    );
+
+    final DruidContainer container = new DruidContainer(command, imageName)
+        .withExposedPorts(exposedPorts.toArray(new Integer[0]))
+        .withCommonProperties(getCommonProperties())
+        .withServiceProperties(getServerProperties())
+        .withFileSystemBind(
+            segmentDeepStorageDirectory.hostPath,
+            segmentDeepStorageDirectory.containerPath,
+            BindMode.READ_WRITE
+        )
+        .withFileSystemBind(
+            indexerLogsDeepStorageDirectory.hostPath,
+            indexerLogsDeepStorageDirectory.containerPath,
+            BindMode.READ_WRITE
+        )
+        .withFileSystemBind(
+            serviceLogsDirectory.hostPath,
+            serviceLogsDirectory.containerPath,
+            BindMode.READ_WRITE
+        )
+        .withEnv(
+            Map.of(
+                "DRUID_SET_HOST_IP", "0",
+                "DRUID_SET_HOST", "0"
+            )
+        )
+        .waitingFor(Wait.forHttp("/status/health").forPort(servicePort));
+
+    // Bind the ports statically (rather than using a mapped port) to the same
+    // value used in `druid.plaintextPort` to make this node discoverable
+    // by other services (both embedded and dockerized).
+    List<String> portBindings = exposedPorts.stream().map(
+        port -> StringUtils.format("%d:%d", port, port)
+    ).collect(Collectors.toList());
+    container.setPortBindings(portBindings);
+
+    return container;
+  }
+
+  /**
+   * Writes the common properties to a file.
+   *
+   * @return Name of the file.
+   */
+  private Properties getCommonProperties()
+  {
+    final Properties commonProperties = new Properties();
+    commonProperties.putAll(cluster.getCommonProperties());
+    FORBIDDEN_PROPERTIES.forEach(commonProperties::remove);
+
+    commonProperties.setProperty(
+        "druid.storage.storageDirectory",
+        segmentDeepStorageDirectory.containerPath
+    );
+    commonProperties.setProperty(
+        "druid.indexer.logs.directory",
+        indexerLogsDeepStorageDirectory.containerPath
+    );
+
+    log.info(
+        "Writing common properties for Druid container[%s]: [%s]",
+        name, commonProperties
+    );
+
+    return commonProperties;
+  }
+
+  /**
+   * Writes the server properties to a file.
+   *
+   * @return Name of the file.
+   */
+  private Properties getServerProperties()
+  {
+    FORBIDDEN_PROPERTIES.forEach(properties::remove);
+    addProperty("druid.host", EmbeddedDruidCluster.getDefaultHost());
+    addProperty("druid.plaintextPort", String.valueOf(servicePort));
+
+    final Properties serverProperties = new Properties();
+    serverProperties.putAll(properties);
+
+    log.info(
+        "Writing runtime properties for Druid container[%s]: [%s]",
+        name, serverProperties
+    );
+
+    return serverProperties;
+  }
+
+  private static void createLogDirectory(File dir)
+  {
+    try {
+      FileUtils.mkdirp(dir);
+      Files.setPosixFilePermissions(dir.toPath(), PosixFilePermissions.fromString("rwxrwxrwx"));
+    }
+    catch (Exception e) {
+      throw new RuntimeException(e);
+    }
+  }

Review Comment:
   couldn't the `MountedDir` provide a place for this method? that way if someone needs similar in the future it might got reused...



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.

kfaraz commented on code in PR #18302:
URL: https://github.com/apache/druid/pull/18302#discussion_r2236060811


##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/DruidContainerResource.java:
##########
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import org.apache.druid.java.util.common.FileUtils;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.testing.DruidCommand;
+import org.apache.druid.testing.DruidContainer;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.TestcontainerResource;
+import org.testcontainers.containers.BindMode;
+import org.testcontainers.containers.wait.strategy.Wait;
+import org.testcontainers.utility.DockerImageName;
+
+import java.io.File;
+import java.nio.file.Files;
+import java.nio.file.attribute.PosixFilePermissions;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+
+/**
+ * {@link TestcontainerResource} to run Druid services.
+ * Currently, only core extensions can be used out-of-the-box with these containers
+ * such as {@code druid-s3-extensions} or {@code postgresql-metadata-storage},
+ * simply by adding them to {@code druid.extensions.loadList}.
+ * <p>
+ * {@link DruidContainers} should be used only for testing backward compatiblity
+ * or a Docker-specific feature. For all other testing needs, use plain old
+ * {@code EmbeddedDruidServer} as they are much faster, allow easy debugging and
+ * do not require downloading any images.
+ */
+public class DruidContainerResource extends TestcontainerResource<DruidContainer>
+{
+  /**
+   * Java system property to specify the name of the Docker test image.
+   */
+  public static final String PROPERTY_TEST_IMAGE = "druid.testing.docker.image";
+
+  private static final Logger log = new Logger(DruidContainerResource.class);
+
+  /**
+   * Forbidden server properties that may be used by EmbeddedDruidServers but
+   * interfere with the functioning of DruidContainer-based services.
+   */
+  private static final Set<String> FORBIDDEN_PROPERTIES = Set.of(
+      "druid.extensions.modulesForEmbeddedTests",
+      "druid.emitter"
+  );
+
+  /**
+   * A static incremental ID is used instead of a random number to ensure that
+   * tests are more deterministic and easier to debug.
+   */
+  private static final AtomicInteger SERVER_ID = new AtomicInteger(0);
+
+  private final String name;
+  private final DruidCommand command;
+  private final Map<String, String> properties = new HashMap<>();
+  private final List<Integer> taskPorts = new ArrayList<>();
+
+  private int servicePort;
+  private DockerImageName imageName;
+  private EmbeddedDruidCluster cluster;
+
+  private String containerDirectory;
+
+  private MountedDir indexerLogsDeepStorageDirectory;
+  private MountedDir serviceLogsDirectory;
+  private MountedDir segmentDeepStorageDirectory;
+
+  DruidContainerResource(DruidCommand command)
+  {
+    this.name = StringUtils.format(
+        "container_%s_%d",

Review Comment:
   This will only ever be used with the embedded test framework. So, I am not sure if included an `embedded` or similar prefix would add much value either.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.

kfaraz commented on code in PR #18302:
URL: https://github.com/apache/druid/pull/18302#discussion_r2236845632


##########
.github/workflows/docker-tests.yml:
##########
@@ -0,0 +1,97 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: "Docker Tests using Distribution Image"
+on:
+  workflow_call:
+
+jobs:
+  run-docker-tests:
+    name: Run Docker tests
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Retrieve Docker container
+        id: docker-restore
+        uses: actions/cache/restore@v4
+        with:
+          key: druid-dist-container.tar.gz-${{ github.sha }}
+          path: |
+            ./druid-dist-container.tar.gz
+      - name: Stop and remove Druid Docker containers
+        run: |
+          echo "Force stopping all Druid containers and pruning"
+          docker ps -aq --filter "ancestor=apache/druid" | xargs -r docker rm -f
+          docker system prune -af --volumes
+      - name: Load Docker image
+        run: |
+          docker load --input druid-dist-container.tar.gz
+          docker images
+      - name: Setup Java
+        uses: actions/setup-java@v4
+        with:
+          distribution: 'zulu'
+          java-version: 17
+          cache: 'maven'
+      - name: Run *DockerTest*
+        id: run-it
+        run: |
+          source .github/scripts/distribution_checks_env.sh
+          .github/scripts/run_unit-tests -Dtest=*DockerTest* -Ddruid.testing.docker.image=$DRUID_DIST_IMAGE_NAME
+        timeout-minutes: 60
+
+      - name: Collect docker logs on failure
+        if: ${{ failure() && steps.run-it.conclusion == 'failure' }}
+        run: |
+          mkdir docker-logs
+          for c in $(docker ps -a --format="{{.Names}}")
+          do
+            docker logs $c > ./docker-logs/$c.log
+          done
+
+      - name: Tar docker logs
+        if: ${{ failure() && steps.run-it.conclusion == 'failure' }}
+        run: tar cvzf ./docker-logs.tgz ./docker-logs
+
+      - name: Upload docker logs to GitHub
+        if: ${{ failure() && steps.run-it.conclusion == 'failure' }}
+        uses: actions/upload-artifact@v4
+        with:
+          name: failure-docker-logs
+          path: docker-logs.tgz
+
+      - name: Collect service logs on failure
+        if: ${{ failure() && steps.run-it.conclusion == 'failure' }}
+        run: |
+          tar cvzf ./service-logs.tgz /tmp/embedded-tests

Review Comment:
   Oh, I think this is not needed anymore. I had used it while debugging the workflow.
   Service logs are written inside a fresh `TestFolder` created by the embedded cluster.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.

kfaraz commented on PR #18302:
URL: https://github.com/apache/druid/pull/18302#issuecomment-3164996113

   Thanks for the reviews, @kgyrtkirk , @capistrant , @gianm !
   Thanks for the assist with the CI, @Akshat-Jain !


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "capistrant (via GitHub)" <gi...@apache.org>.

capistrant commented on code in PR #18302:
URL: https://github.com/apache/druid/pull/18302#discussion_r2258458691


##########
services/src/test/java/org/apache/druid/testing/embedded/EmbeddedDruidCluster.java:
##########
@@ -232,13 +228,21 @@ public TestFolder getTestFolder()
   }
 
   /**
-   * The embedded Zookeeper server used by this cluster, if any.
-   *
-   * @throws NullPointerException if this cluster has no embedded zookeeper.
+   * Uses a container-friendly hostname for all embedded services, Druid as well
+   * as external.
+   */
+  public EmbeddedDruidCluster useContainerFriendlyHostname()

Review Comment:
   ya, I agree. I'm probably just overthinking it since I'm currently working on the KafkaDataFormat tests which requires an extra docker container that can talk to the kafka container :) 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.

kfaraz commented on code in PR #18302:
URL: https://github.com/apache/druid/pull/18302#discussion_r2236189665


##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/DruidContainers.java:
##########
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import org.apache.druid.testing.DruidCommand;
+
+/**
+ * Factory for {@link DruidContainerResource} that can run specific services.
+ *
+ * @see #newOverlord()
+ * @see #newCoordinator()
+ */
+public final class DruidContainers
+{
+  private DruidContainers()
+  {
+    // no instantiation
+  }
+
+  /**
+   * Creates a new {@link DruidContainerResource} to run a Coordinator node.
+   */
+  public static DruidContainerResource newCoordinator()
+  {
+    return new DruidContainerResource(DruidCommand.COORDINATOR)
+        .addProperty("druid.coordinator.startDelay", "PT0.1S")
+        .addProperty("druid.coordinator.period", "PT0.5S")
+        .addProperty("druid.manager.segments.pollDuration", "PT0.1S");
+  }
+
+  /**
+   * Creates a new {@link DruidContainerResource} to run an Overlord node.
+   */
+  public static DruidContainerResource newOverlord()
+  {
+    // Keep a small sync timeout so that Peons and Indexers are not stuck
+    // handling a change request when Overlord has already shutdown
+    return new DruidContainerResource(DruidCommand.OVERLORD)
+        .addProperty("druid.indexer.storage.type", "metadata")
+        .addProperty("druid.indexer.queue.startDelay", "PT0S")
+        .addProperty("druid.indexer.queue.restartDelay", "PT0S")
+        .addProperty("druid.indexer.runner.syncRequestTimeout", "PT1S");
+  }
+
+  /**
+   * Creates a new {@link DruidContainerResource} to run an Indexer node.
+   */
+  public static DruidContainerResource newIndexer()
+  {
+    return new DruidContainerResource(DruidCommand.INDEXER)
+        .addProperty("druid.lookup.enableLookupSyncOnStartup", "false")
+        .addProperty("druid.processing.buffer.sizeBytes", "50MiB")
+        .addProperty("druid.processing.numMergeBuffers", "2")
+        .addProperty("druid.processing.numThreads", "5");
+  }

Review Comment:
   Some of these properties might make it into DruidCommand itself if we feel that these are always needed out of the box whenever running a DruidContainer.
   
   I have just kept the minimal set in DruidCommand for now but will double check to confirm if some of these can't be moved there right now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.

kfaraz commented on code in PR #18302:
URL: https://github.com/apache/druid/pull/18302#discussion_r2236977856


##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/DruidContainers.java:
##########
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import org.apache.druid.testing.DruidCommand;
+
+/**
+ * Factory for {@link DruidContainerResource} that can run specific services.
+ *
+ * @see #newOverlord()
+ * @see #newCoordinator()
+ */
+public final class DruidContainers
+{
+  private DruidContainers()
+  {
+    // no instantiation
+  }
+
+  /**
+   * Creates a new {@link DruidContainerResource} to run a Coordinator node.
+   */

Review Comment:
   Updated.



##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/DruidContainerResource.java:
##########
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import org.apache.druid.java.util.common.FileUtils;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.testing.DruidCommand;
+import org.apache.druid.testing.DruidContainer;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.TestcontainerResource;
+import org.testcontainers.containers.BindMode;
+import org.testcontainers.containers.wait.strategy.Wait;
+import org.testcontainers.utility.DockerImageName;
+
+import java.io.File;
+import java.nio.file.Files;
+import java.nio.file.attribute.PosixFilePermissions;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+
+/**
+ * {@link TestcontainerResource} to run Druid services.
+ * Currently, only core extensions can be used out-of-the-box with these containers
+ * such as {@code druid-s3-extensions} or {@code postgresql-metadata-storage},
+ * simply by adding them to {@code druid.extensions.loadList}.
+ * <p>
+ * {@link DruidContainers} should be used only for testing backward compatiblity
+ * or a Docker-specific feature. For all other testing needs, use plain old
+ * {@code EmbeddedDruidServer} as they are much faster, allow easy debugging and
+ * do not require downloading any images.
+ */
+public class DruidContainerResource extends TestcontainerResource<DruidContainer>
+{
+  /**
+   * Java system property to specify the name of the Docker test image.
+   */
+  public static final String PROPERTY_TEST_IMAGE = "druid.testing.docker.image";
+
+  private static final Logger log = new Logger(DruidContainerResource.class);
+
+  /**
+   * Forbidden server properties that may be used by EmbeddedDruidServers but
+   * interfere with the functioning of DruidContainer-based services.
+   */
+  private static final Set<String> FORBIDDEN_PROPERTIES = Set.of(
+      "druid.extensions.modulesForEmbeddedTests",
+      "druid.emitter"
+  );
+
+  /**
+   * A static incremental ID is used instead of a random number to ensure that
+   * tests are more deterministic and easier to debug.
+   */
+  private static final AtomicInteger SERVER_ID = new AtomicInteger(0);
+
+  private final String name;
+  private final DruidCommand command;
+  private final Map<String, String> properties = new HashMap<>();
+  private final List<Integer> taskPorts = new ArrayList<>();
+
+  private int servicePort;
+  private DockerImageName imageName;
+  private EmbeddedDruidCluster cluster;
+
+  private String containerDirectory;
+
+  private MountedDir indexerLogsDeepStorageDirectory;
+  private MountedDir serviceLogsDirectory;
+  private MountedDir segmentDeepStorageDirectory;
+
+  DruidContainerResource(DruidCommand command)
+  {
+    this.name = StringUtils.format(
+        "container_%s_%d",
+        command.getName(),
+        SERVER_ID.incrementAndGet()
+    );
+    this.command = command;
+
+    Integer[] exposedPorts = command.getExposedPorts();
+    servicePort = exposedPorts[0];
+    taskPorts.addAll(Arrays.asList(exposedPorts).subList(1, exposedPorts.length));
+  }
+
+  public DruidContainerResource withPort(int port)
+  {
+    this.servicePort = port;
+    return this;
+  }
+
+  /**
+   * Used to bind the task ports on middle managers.
+   */
+  public DruidContainerResource withTaskPorts(int startPort, int workerCapacity)
+  {
+    taskPorts.clear();
+    for (int i = 0; i < workerCapacity; ++i) {
+      taskPorts.add(startPort + i);
+    }
+    addProperty("druid.worker.capacity", String.valueOf(workerCapacity));
+    addProperty("druid.indexer.runner.startPort", String.valueOf(startPort));
+    addProperty("druid.indexer.runner.endPort", String.valueOf(startPort + workerCapacity));
+
+    return this;
+  }
+
+  public DruidContainerResource withImage(DockerImageName imageName)
+  {
+    this.imageName = imageName;
+    return this;
+  }
+
+  /**
+   * Uses the Docker test image specified by the system property
+   * {@link #PROPERTY_TEST_IMAGE} for this container.
+   */
+  public DruidContainerResource withTestImage()
+  {
+    String imageName = Objects.requireNonNull(
+        System.getProperty(PROPERTY_TEST_IMAGE),
+        StringUtils.format("System property[%s] is not set", PROPERTY_TEST_IMAGE)
+    );
+    return withImage(DockerImageName.parse(imageName));
+  }
+
+  public DruidContainerResource addProperty(String key, String value)
+  {
+    properties.put(key, value);
+    return this;
+  }
+
+  @Override
+  public void beforeStart(EmbeddedDruidCluster cluster)
+  {
+    this.cluster = cluster;
+
+    // Mount directories used by the entire cluster (including embedded servers)
+    this.segmentDeepStorageDirectory = new MountedDir(
+        "/tmp/druid/deep-store",
+        cluster.getTestFolder().getOrCreateFolder("deep-store").getAbsolutePath()
+    );
+    this.indexerLogsDeepStorageDirectory = new MountedDir(
+        "/tmp/druid/indexer-logs",
+        cluster.getTestFolder().getOrCreateFolder("indexer-logs").getAbsolutePath()
+    );
+
+    // Mount directories used by this container for easier debugging with service logs
+    this.containerDirectory = cluster.getTestFolder().getOrCreateFolder(name).getAbsolutePath();
+    this.serviceLogsDirectory = new MountedDir(
+        "/opt/druid/log",
+        containerDirectory + "/log"
+    );
+
+    // Create the log directory upfront to avoid permission issues
+    createLogDirectory(new File(containerDirectory, "log"));
+  }
+
+  @Override
+  protected DruidContainer createContainer()
+  {
+    final List<Integer> exposedPorts = new ArrayList<>(taskPorts);
+    exposedPorts.add(servicePort);
+    log.info(
+        "Starting Druid container[%s] on port[%d] with mounted directory[%s] and exposed ports[%s].",
+        name, servicePort, containerDirectory, exposedPorts
+    );
+
+    final DruidContainer container = new DruidContainer(command, imageName)
+        .withExposedPorts(exposedPorts.toArray(new Integer[0]))
+        .withCommonProperties(getCommonProperties())
+        .withServiceProperties(getServerProperties())
+        .withFileSystemBind(
+            segmentDeepStorageDirectory.hostPath,
+            segmentDeepStorageDirectory.containerPath,
+            BindMode.READ_WRITE
+        )
+        .withFileSystemBind(
+            indexerLogsDeepStorageDirectory.hostPath,
+            indexerLogsDeepStorageDirectory.containerPath,
+            BindMode.READ_WRITE
+        )
+        .withFileSystemBind(
+            serviceLogsDirectory.hostPath,
+            serviceLogsDirectory.containerPath,
+            BindMode.READ_WRITE
+        )
+        .withEnv(
+            Map.of(
+                "DRUID_SET_HOST_IP", "0",
+                "DRUID_SET_HOST", "0"
+            )
+        )
+        .waitingFor(Wait.forHttp("/status/health").forPort(servicePort));
+
+    // Bind the ports statically (rather than using a mapped port) to the same
+    // value used in `druid.plaintextPort` to make this node discoverable
+    // by other services (both embedded and dockerized).
+    List<String> portBindings = exposedPorts.stream().map(
+        port -> StringUtils.format("%d:%d", port, port)
+    ).collect(Collectors.toList());
+    container.setPortBindings(portBindings);
+
+    return container;
+  }
+
+  /**
+   * Writes the common properties to a file.
+   *
+   * @return Name of the file.
+   */
+  private Properties getCommonProperties()
+  {
+    final Properties commonProperties = new Properties();
+    commonProperties.putAll(cluster.getCommonProperties());
+    FORBIDDEN_PROPERTIES.forEach(commonProperties::remove);
+
+    commonProperties.setProperty(
+        "druid.storage.storageDirectory",
+        segmentDeepStorageDirectory.containerPath
+    );
+    commonProperties.setProperty(
+        "druid.indexer.logs.directory",
+        indexerLogsDeepStorageDirectory.containerPath
+    );
+
+    log.info(
+        "Writing common properties for Druid container[%s]: [%s]",
+        name, commonProperties
+    );
+
+    return commonProperties;
+  }
+
+  /**
+   * Writes the server properties to a file.
+   *
+   * @return Name of the file.
+   */

Review Comment:
   updated.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.

kfaraz commented on code in PR #18302:
URL: https://github.com/apache/druid/pull/18302#discussion_r2236986488


##########
.github/scripts/distribution_checks_env.sh:
##########
@@ -0,0 +1,20 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+export DRUID_DIST_IMAGE_NAME=apache/druid:35.0.0-SNAPSHOT

Review Comment:
   removed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "github-advanced-security[bot] (via GitHub)" <gi...@apache.org>.

github-advanced-security[bot] commented on code in PR #18302:
URL: https://github.com/apache/druid/pull/18302#discussion_r2219013921


##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/IngestionDockerTest.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.druid.common.utils.IdUtils;
+import org.apache.druid.data.input.impl.CsvInputFormat;
+import org.apache.druid.data.input.impl.DimensionsSpec;
+import org.apache.druid.data.input.impl.TimestampSpec;
+import org.apache.druid.indexer.TaskState;
+import org.apache.druid.indexer.TaskStatusPlus;
+import org.apache.druid.indexing.kafka.KafkaIndexTaskModule;
+import org.apache.druid.indexing.kafka.simulate.KafkaResource;
+import org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorIOConfig;
+import org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorSpec;
+import org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorTuningConfig;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.supervisor.SupervisorStatus;
+import org.apache.druid.java.util.common.DateTimes;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.parsers.CloseableIterator;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.indexing.DataSchema;
+import org.apache.druid.testing.embedded.EmbeddedBroker;
+import org.apache.druid.testing.embedded.EmbeddedClusterApis;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.EmbeddedOverlord;
+import org.apache.druid.testing.embedded.EmbeddedRouter;
+import org.apache.druid.testing.embedded.derby.EmbeddedDerbyMetadataResource;
+import org.apache.druid.testing.embedded.indexing.Resources;
+import org.apache.druid.testing.embedded.junit5.EmbeddedClusterTestBase;
+import org.apache.druid.testing.embedded.minio.MinIOStorageResource;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.joda.time.DateTime;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ThreadLocalRandom;
+
+/**
+ * Runs some basic ingestion tests using {@link DruidContainers}.
+ */
+public class IngestionDockerTest extends EmbeddedClusterTestBase
+{
+  static {
+    System.setProperty("druid.testing.docker.image", "apache/druid:35.0.0-SNAPSHOT");
+  }
+
+  // Druid Docker containers
+  protected final DruidContainerResource overlordLeader = DruidContainers.newOverlord().withTestImage();
+  protected final DruidContainerResource coordinator = DruidContainers.newCoordinator().withTestImage();
+  protected final DruidContainerResource historical = DruidContainers.newHistorical().withTestImage();
+  protected final DruidContainerResource broker1 = DruidContainers.newBroker().withTestImage();
+  protected final DruidContainerResource middleManager = DruidContainers
+      .newMiddleManager()
+      .withTestImage()
+      .addProperty("druid.segment.handoff.pollDuration", "PT0.1s");
+
+  // Follower EmbeddedOverlord to watch segment publish events
+  private final EmbeddedOverlord overlordFollower = new EmbeddedOverlord()
+      .addProperty("druid.plaintextPort", "7090")
+      .addProperty("druid.manager.segments.useIncrementalCache", "always")
+      .addProperty("druid.manager.segments.pollDuration", "PT0.1s");
+
+  // Additional EmbeddedBroker to wait for segments to become queryable
+  private final EmbeddedBroker broker2 = new EmbeddedBroker()
+      .addProperty("druid.plaintextPort", "7082");
+
+  private final KafkaResource kafkaServer = new KafkaResource();
+
+  @Override
+  public EmbeddedDruidCluster createCluster()
+  {
+    final EmbeddedDruidCluster cluster = EmbeddedDruidCluster.withZookeeper();
+
+    return cluster
+        .useDruidContainers()
+        .useLatchableEmitter()
+        // Needed for overlordFollower to recognize the KafkaSupervisor type
+        .addExtension(KafkaIndexTaskModule.class)
+        .addResource(new EmbeddedDerbyMetadataResource())
+        .addResource(new MinIOStorageResource())
+        .addResource(kafkaServer)
+        .addCommonProperty(
+            "druid.extensions.loadList",
+            "[\"druid-s3-extensions\", \"druid-kafka-indexing-service\", \"druid-multi-stage-query\"]"
+        )
+        .addResource(coordinator)
+        .addResource(overlordLeader)
+        .addResource(middleManager)
+        .addResource(historical)
+        .addResource(broker1)
+        .addServer(overlordFollower)
+        .addServer(broker2)
+        .addServer(new EmbeddedRouter());
+  }
+
+  @Test
+  public void test_runIndexTask_andKillData()
+  {
+
+  }
+
+  @Test
+  public void test_runIndexParallelTask_andCompactData()
+  {
+
+  }
+
+  @Test
+  public void test_runMsqTask_andQueryData()
+  {
+
+  }
+
+  @Test
+  public void test_runIndexTask()
+  {
+    final String taskId = IdUtils.getRandomId();
+    final Object task = createIndexTaskForInlineData(
+        taskId,
+        StringUtils.replace(Resources.CSV_DATA_10_DAYS, "\n", "\\n")
+    );
+
+    cluster.callApi().onLeaderOverlord(o -> o.runTask(taskId, task));
+    waitForCachedUsedSegmentCount(10);
+    verifyUsedSegmentCount(10);
+  }
+
+  private Object createIndexTaskForInlineData(String taskId, String inlineDataCsv)
+  {
+    return EmbeddedClusterApis.createTaskFromPayload(
+        taskId,
+        StringUtils.format(Resources.INDEX_TASK_PAYLOAD_WITH_INLINE_DATA, inlineDataCsv, dataSource)
+    );
+  }
+
+  @Test
+  public void test_runKafkaSupervisor()
+  {
+    final String topic = dataSource;
+    kafkaServer.createTopicWithPartitions(topic, 2);
+
+    kafkaServer.produceRecordsToTopic(
+        generateRecordsForTopic(topic, 10, DateTimes.of("2025-06-01"))
+    );
+
+    // Submit and start a supervisor
+    final String supervisorId = dataSource;
+    final KafkaSupervisorSpec kafkaSupervisorSpec = createKafkaSupervisor(supervisorId, topic);
+
+    final Map<String, String> startSupervisorResult = cluster.callApi().onLeaderOverlord(
+        o -> o.postSupervisor(kafkaSupervisorSpec)
+    );
+    Assertions.assertEquals(Map.of("id", supervisorId), startSupervisorResult);
+
+    // Wait for the broker to discover the realtime segments
+    broker2.latchableEmitter().waitForEvent(
+        event -> event.hasDimension(DruidMetrics.DATASOURCE, dataSource)
+    );
+
+    SupervisorStatus supervisorStatus = cluster.callApi().getSupervisorStatus(supervisorId);
+    Assertions.assertFalse(supervisorStatus.isSuspended());
+    Assertions.assertTrue(supervisorStatus.isHealthy());
+    Assertions.assertEquals("RUNNING", supervisorStatus.getState());
+    Assertions.assertEquals(topic, supervisorStatus.getSource());
+
+    // Get the task statuses
+    List<TaskStatusPlus> taskStatuses = ImmutableList.copyOf(
+        (CloseableIterator<TaskStatusPlus>)
+            cluster.callApi().onLeaderOverlord(o -> o.taskStatuses(null, dataSource, 1))
+    );
+    Assertions.assertEquals(1, taskStatuses.size());
+    Assertions.assertEquals(TaskState.RUNNING, taskStatuses.get(0).getStatusCode());
+
+    // Verify the count of rows ingested into the datasource so far
+    // Assertions.assertEquals("10", cluster.runSql("SELECT COUNT(*) FROM %s", dataSource));
+
+    // Suspend the supervisor and verify the state
+    cluster.callApi().onLeaderOverlord(
+        o -> o.postSupervisor(kafkaSupervisorSpec.createSuspendedSpec())
+    );
+    supervisorStatus = cluster.callApi().getSupervisorStatus(supervisorId);
+    Assertions.assertTrue(supervisorStatus.isSuspended());
+
+    // waitForCachedUsedSegmentCount(10);

Review Comment:
   ## Useless parameter
   
   The parameter 'supervisorId' is never used.
   
   [Show more details](https://github.com/apache/druid/security/code-scanning/9901)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.

kfaraz commented on code in PR #18302:
URL: https://github.com/apache/druid/pull/18302#discussion_r2238957015


##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/IngestionDockerTest.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.druid.common.utils.IdUtils;
+import org.apache.druid.data.input.impl.CsvInputFormat;
+import org.apache.druid.data.input.impl.DimensionsSpec;
+import org.apache.druid.data.input.impl.TimestampSpec;
+import org.apache.druid.indexer.TaskState;
+import org.apache.druid.indexer.TaskStatusPlus;
+import org.apache.druid.indexing.kafka.KafkaIndexTaskModule;
+import org.apache.druid.indexing.kafka.simulate.KafkaResource;
+import org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorIOConfig;
+import org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorSpec;
+import org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorTuningConfig;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.supervisor.SupervisorStatus;
+import org.apache.druid.java.util.common.DateTimes;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.parsers.CloseableIterator;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.indexing.DataSchema;
+import org.apache.druid.testing.embedded.EmbeddedBroker;
+import org.apache.druid.testing.embedded.EmbeddedClusterApis;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.EmbeddedOverlord;
+import org.apache.druid.testing.embedded.EmbeddedRouter;
+import org.apache.druid.testing.embedded.derby.EmbeddedDerbyMetadataResource;
+import org.apache.druid.testing.embedded.indexing.Resources;
+import org.apache.druid.testing.embedded.junit5.EmbeddedClusterTestBase;
+import org.apache.druid.testing.embedded.minio.MinIOStorageResource;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.joda.time.DateTime;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ThreadLocalRandom;
+
+/**
+ * Runs some basic ingestion tests using {@link DruidContainers}.
+ */
+public class IngestionDockerTest extends EmbeddedClusterTestBase
+{
+  static {
+    System.setProperty("druid.testing.docker.image", "apache/druid:35.0.0-SNAPSHOT");

Review Comment:
   > Setting this in a static initializer doesn't seem right, because it will be set when the class is loaded, not necessarily when the test runs. Maybe better to use @BeforeAll.
   
   Sorry, this was not meant to be committed and was for my local testing only.
   Must have crept in in a recent commit.
   
   In CI workflows, this property is always passed as a Java argument `-Ddruid.testing.docker.image`.
   So, it needs to be set only when running the test locally.
   
   I chose purposefully not to use a default image so that each test has to set it explicitly.
   `KafkaContainer` seems to have deprecated the no-arg constructor too, so that test writers explicitly call out the required image name.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "Akshat-Jain (via GitHub)" <gi...@apache.org>.

Akshat-Jain closed pull request #18302: Add DruidContainers to run docker tests with embedded-test framework
URL: https://github.com/apache/druid/pull/18302


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "gianm (via GitHub)" <gi...@apache.org>.

gianm commented on code in PR #18302:
URL: https://github.com/apache/druid/pull/18302#discussion_r2244439241


##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/IngestionDockerTest.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.druid.common.utils.IdUtils;
+import org.apache.druid.data.input.impl.CsvInputFormat;
+import org.apache.druid.data.input.impl.DimensionsSpec;
+import org.apache.druid.data.input.impl.TimestampSpec;
+import org.apache.druid.indexer.TaskState;
+import org.apache.druid.indexer.TaskStatusPlus;
+import org.apache.druid.indexing.kafka.KafkaIndexTaskModule;
+import org.apache.druid.indexing.kafka.simulate.KafkaResource;
+import org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorIOConfig;
+import org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorSpec;
+import org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorTuningConfig;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.supervisor.SupervisorStatus;
+import org.apache.druid.java.util.common.DateTimes;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.parsers.CloseableIterator;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.indexing.DataSchema;
+import org.apache.druid.testing.embedded.EmbeddedBroker;
+import org.apache.druid.testing.embedded.EmbeddedClusterApis;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.EmbeddedOverlord;
+import org.apache.druid.testing.embedded.EmbeddedRouter;
+import org.apache.druid.testing.embedded.derby.EmbeddedDerbyMetadataResource;
+import org.apache.druid.testing.embedded.indexing.Resources;
+import org.apache.druid.testing.embedded.junit5.EmbeddedClusterTestBase;
+import org.apache.druid.testing.embedded.minio.MinIOStorageResource;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.joda.time.DateTime;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ThreadLocalRandom;
+
+/**
+ * Runs some basic ingestion tests using {@link DruidContainers}.
+ */
+public class IngestionDockerTest extends EmbeddedClusterTestBase
+{
+  static {
+    System.setProperty("druid.testing.docker.image", "apache/druid:35.0.0-SNAPSHOT");

Review Comment:
   I wonder if the test should be disabled by default, and enabled using a mvn profile. The case for disabling it by default is that it isn't actually testing the current code in the repository. It's testing a image pulled from somewhere else. Having it be its own command makes it more clear what is happening. During voting if committers want to test the docker image too, that can be done with a special `mvn` command separate from the typical test command.
   
   How does this sound to you?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.

kfaraz commented on code in PR #18302:
URL: https://github.com/apache/druid/pull/18302#discussion_r2245496237


##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/IngestionDockerTest.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.druid.common.utils.IdUtils;
+import org.apache.druid.data.input.impl.CsvInputFormat;
+import org.apache.druid.data.input.impl.DimensionsSpec;
+import org.apache.druid.data.input.impl.TimestampSpec;
+import org.apache.druid.indexer.TaskState;
+import org.apache.druid.indexer.TaskStatusPlus;
+import org.apache.druid.indexing.kafka.KafkaIndexTaskModule;
+import org.apache.druid.indexing.kafka.simulate.KafkaResource;
+import org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorIOConfig;
+import org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorSpec;
+import org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorTuningConfig;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.supervisor.SupervisorStatus;
+import org.apache.druid.java.util.common.DateTimes;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.parsers.CloseableIterator;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.indexing.DataSchema;
+import org.apache.druid.testing.embedded.EmbeddedBroker;
+import org.apache.druid.testing.embedded.EmbeddedClusterApis;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.EmbeddedOverlord;
+import org.apache.druid.testing.embedded.EmbeddedRouter;
+import org.apache.druid.testing.embedded.derby.EmbeddedDerbyMetadataResource;
+import org.apache.druid.testing.embedded.indexing.Resources;
+import org.apache.druid.testing.embedded.junit5.EmbeddedClusterTestBase;
+import org.apache.druid.testing.embedded.minio.MinIOStorageResource;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.joda.time.DateTime;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ThreadLocalRandom;
+
+/**
+ * Runs some basic ingestion tests using {@link DruidContainers}.
+ */
+public class IngestionDockerTest extends EmbeddedClusterTestBase
+{
+  static {
+    System.setProperty("druid.testing.docker.image", "apache/druid:35.0.0-SNAPSHOT");

Review Comment:
   Thanks for the suggestion!
   I agree that this would be the cleanest approach.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.

kfaraz commented on code in PR #18302:
URL: https://github.com/apache/druid/pull/18302#discussion_r2246924174


##########
.github/workflows/ci.yml:
##########
@@ -27,7 +27,7 @@ jobs:
         pattern: [ "C*", "H*,U*,V*", "N*,Q*,S*", "B*,O*,R*", "G*,J*,K*", "F*,L*,M*", "A*,D*,I*,X*,Y*,Z*", "E*,P*,T*,W*"]
     uses: ./.github/workflows/worker.yml
     with:
-      script: .github/scripts/run_unit-tests -Dtest=!QTest,'${{ matrix.pattern }}' -Dmaven.test.failure.ignore=true
+      script: .github/scripts/run_unit-tests -Dtest=!QTest,!*DockerTest*,'${{ matrix.pattern }}' -Dmaven.test.failure.ignore=true

Review Comment:
   Fixed. Please check the latest comments.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.

kfaraz commented on code in PR #18302:
URL: https://github.com/apache/druid/pull/18302#discussion_r2257582882


##########
services/src/test/java/org/apache/druid/testing/embedded/EmbeddedDruidCluster.java:
##########
@@ -232,13 +228,21 @@ public TestFolder getTestFolder()
   }
 
   /**
-   * The embedded Zookeeper server used by this cluster, if any.
-   *
-   * @throws NullPointerException if this cluster has no embedded zookeeper.
+   * Uses a container-friendly hostname for all embedded services, Druid as well
+   * as external.
+   */
+  public EmbeddedDruidCluster useContainerFriendlyHostname()

Review Comment:
   The current default (i.e. localhost) should already be good enough for all (non container) embedded tests.
   
   For Druid container based tests, users writing new tests need not be conscious of the choice as long as they extend `DockerTestBase`.
   Also, since Druid containers are not meant to be used frequently except in a couple of smoke tests, the extra step is probably okay?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "gianm (via GitHub)" <gi...@apache.org>.

gianm commented on code in PR #18302:
URL: https://github.com/apache/druid/pull/18302#discussion_r2259803379


##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/IngestionBackwardCompatibilityDockerTest.java:
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.druid.client.coordinator.CoordinatorServiceClient;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.http.client.response.BytesFullResponseHandler;
+import org.apache.druid.java.util.http.client.response.BytesFullResponseHolder;
+import org.apache.druid.rpc.RequestBuilder;
+import org.apache.druid.rpc.indexing.SegmentUpdateResponse;
+import org.apache.druid.testing.DruidContainer;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.jboss.netty.handler.codec.http.HttpMethod;
+import org.junit.jupiter.api.Nested;
+
+/**
+ * Runs some basic ingestion tests using {@code DruidContainers} to verify
+ * backward compatibility with old Druid images.
+ */
+public class IngestionBackwardCompatibilityDockerTest
+{
+  @Nested
+  public class Apache31 extends IngestionDockerTest
+  {
+    @Override
+    public EmbeddedDruidCluster createCluster()
+    {
+      coordinator.usingImage(DruidContainer.Image.APACHE_31);

Review Comment:
   > I agree. In the current code, I am using the `@Tag("docker-tests")` annotation (applied on `DockerTestBase`) to skip the Docker-based tests.
   > The `*DockerTest*` class name is to tell `mvn verify` to look for these classes while searching for eligible integration tests. Otherwise, it only looks for `*IT` or `IT*` classes by default.
   
   Ah, I see. I missed that when reading the maven stuff.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.

kfaraz commented on code in PR #18302:
URL: https://github.com/apache/druid/pull/18302#discussion_r2236065662


##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/DruidContainerResource.java:
##########
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import org.apache.druid.java.util.common.FileUtils;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.testing.DruidCommand;
+import org.apache.druid.testing.DruidContainer;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.TestcontainerResource;
+import org.testcontainers.containers.BindMode;
+import org.testcontainers.containers.wait.strategy.Wait;
+import org.testcontainers.utility.DockerImageName;
+
+import java.io.File;
+import java.nio.file.Files;
+import java.nio.file.attribute.PosixFilePermissions;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+
+/**
+ * {@link TestcontainerResource} to run Druid services.
+ * Currently, only core extensions can be used out-of-the-box with these containers
+ * such as {@code druid-s3-extensions} or {@code postgresql-metadata-storage},
+ * simply by adding them to {@code druid.extensions.loadList}.
+ * <p>
+ * {@link DruidContainers} should be used only for testing backward compatiblity
+ * or a Docker-specific feature. For all other testing needs, use plain old
+ * {@code EmbeddedDruidServer} as they are much faster, allow easy debugging and
+ * do not require downloading any images.
+ */
+public class DruidContainerResource extends TestcontainerResource<DruidContainer>
+{
+  /**
+   * Java system property to specify the name of the Docker test image.
+   */
+  public static final String PROPERTY_TEST_IMAGE = "druid.testing.docker.image";
+
+  private static final Logger log = new Logger(DruidContainerResource.class);
+
+  /**
+   * Forbidden server properties that may be used by EmbeddedDruidServers but
+   * interfere with the functioning of DruidContainer-based services.
+   */
+  private static final Set<String> FORBIDDEN_PROPERTIES = Set.of(
+      "druid.extensions.modulesForEmbeddedTests",
+      "druid.emitter"
+  );
+
+  /**
+   * A static incremental ID is used instead of a random number to ensure that
+   * tests are more deterministic and easier to debug.
+   */
+  private static final AtomicInteger SERVER_ID = new AtomicInteger(0);
+
+  private final String name;
+  private final DruidCommand command;
+  private final Map<String, String> properties = new HashMap<>();
+  private final List<Integer> taskPorts = new ArrayList<>();
+
+  private int servicePort;
+  private DockerImageName imageName;
+  private EmbeddedDruidCluster cluster;
+
+  private String containerDirectory;
+
+  private MountedDir indexerLogsDeepStorageDirectory;
+  private MountedDir serviceLogsDirectory;
+  private MountedDir segmentDeepStorageDirectory;
+
+  DruidContainerResource(DruidCommand command)
+  {
+    this.name = StringUtils.format(
+        "container_%s_%d",
+        command.getName(),
+        SERVER_ID.incrementAndGet()
+    );
+    this.command = command;
+
+    Integer[] exposedPorts = command.getExposedPorts();
+    servicePort = exposedPorts[0];
+    taskPorts.addAll(Arrays.asList(exposedPorts).subList(1, exposedPorts.length));
+  }
+
+  public DruidContainerResource withPort(int port)
+  {
+    this.servicePort = port;
+    return this;
+  }
+
+  /**
+   * Used to bind the task ports on middle managers.
+   */
+  public DruidContainerResource withTaskPorts(int startPort, int workerCapacity)
+  {
+    taskPorts.clear();
+    for (int i = 0; i < workerCapacity; ++i) {
+      taskPorts.add(startPort + i);
+    }
+    addProperty("druid.worker.capacity", String.valueOf(workerCapacity));
+    addProperty("druid.indexer.runner.startPort", String.valueOf(startPort));
+    addProperty("druid.indexer.runner.endPort", String.valueOf(startPort + workerCapacity));
+
+    return this;
+  }
+
+  public DruidContainerResource withImage(DockerImageName imageName)
+  {
+    this.imageName = imageName;
+    return this;
+  }
+
+  /**
+   * Uses the Docker test image specified by the system property
+   * {@link #PROPERTY_TEST_IMAGE} for this container.
+   */
+  public DruidContainerResource withTestImage()
+  {
+    String imageName = Objects.requireNonNull(
+        System.getProperty(PROPERTY_TEST_IMAGE),
+        StringUtils.format("System property[%s] is not set", PROPERTY_TEST_IMAGE)
+    );
+    return withImage(DockerImageName.parse(imageName));
+  }
+
+  public DruidContainerResource addProperty(String key, String value)
+  {
+    properties.put(key, value);
+    return this;
+  }
+
+  @Override
+  public void beforeStart(EmbeddedDruidCluster cluster)
+  {
+    this.cluster = cluster;
+
+    // Mount directories used by the entire cluster (including embedded servers)
+    this.segmentDeepStorageDirectory = new MountedDir(
+        "/tmp/druid/deep-store",
+        cluster.getTestFolder().getOrCreateFolder("deep-store").getAbsolutePath()
+    );
+    this.indexerLogsDeepStorageDirectory = new MountedDir(
+        "/tmp/druid/indexer-logs",
+        cluster.getTestFolder().getOrCreateFolder("indexer-logs").getAbsolutePath()
+    );
+
+    // Mount directories used by this container for easier debugging with service logs
+    this.containerDirectory = cluster.getTestFolder().getOrCreateFolder(name).getAbsolutePath();
+    this.serviceLogsDirectory = new MountedDir(
+        "/opt/druid/log",
+        containerDirectory + "/log"
+    );
+
+    // Create the log directory upfront to avoid permission issues
+    createLogDirectory(new File(containerDirectory, "log"));
+  }
+
+  @Override
+  protected DruidContainer createContainer()
+  {
+    final List<Integer> exposedPorts = new ArrayList<>(taskPorts);
+    exposedPorts.add(servicePort);
+    log.info(
+        "Starting Druid container[%s] on port[%d] with mounted directory[%s] and exposed ports[%s].",
+        name, servicePort, containerDirectory, exposedPorts
+    );
+
+    final DruidContainer container = new DruidContainer(command, imageName)
+        .withExposedPorts(exposedPorts.toArray(new Integer[0]))

Review Comment:
   Felt unnecessary since this is just used in one place here.
   These are methods exposed by `GenericContainer` from `Testcontainers` library, not added by this PR.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.

kfaraz commented on code in PR #18302:
URL: https://github.com/apache/druid/pull/18302#discussion_r2236189665


##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/DruidContainers.java:
##########
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import org.apache.druid.testing.DruidCommand;
+
+/**
+ * Factory for {@link DruidContainerResource} that can run specific services.
+ *
+ * @see #newOverlord()
+ * @see #newCoordinator()
+ */
+public final class DruidContainers
+{
+  private DruidContainers()
+  {
+    // no instantiation
+  }
+
+  /**
+   * Creates a new {@link DruidContainerResource} to run a Coordinator node.
+   */
+  public static DruidContainerResource newCoordinator()
+  {
+    return new DruidContainerResource(DruidCommand.COORDINATOR)
+        .addProperty("druid.coordinator.startDelay", "PT0.1S")
+        .addProperty("druid.coordinator.period", "PT0.5S")
+        .addProperty("druid.manager.segments.pollDuration", "PT0.1S");
+  }
+
+  /**
+   * Creates a new {@link DruidContainerResource} to run an Overlord node.
+   */
+  public static DruidContainerResource newOverlord()
+  {
+    // Keep a small sync timeout so that Peons and Indexers are not stuck
+    // handling a change request when Overlord has already shutdown
+    return new DruidContainerResource(DruidCommand.OVERLORD)
+        .addProperty("druid.indexer.storage.type", "metadata")
+        .addProperty("druid.indexer.queue.startDelay", "PT0S")
+        .addProperty("druid.indexer.queue.restartDelay", "PT0S")
+        .addProperty("druid.indexer.runner.syncRequestTimeout", "PT1S");
+  }
+
+  /**
+   * Creates a new {@link DruidContainerResource} to run an Indexer node.
+   */
+  public static DruidContainerResource newIndexer()
+  {
+    return new DruidContainerResource(DruidCommand.INDEXER)
+        .addProperty("druid.lookup.enableLookupSyncOnStartup", "false")
+        .addProperty("druid.processing.buffer.sizeBytes", "50MiB")
+        .addProperty("druid.processing.numMergeBuffers", "2")
+        .addProperty("druid.processing.numThreads", "5");
+  }

Review Comment:
   Some of these properties might make it into DruidCommand itself if we feel that these are always needed out of the box whenever running a DruidContainer.
   
   I have just kept the minimal set in DruidCommand for now but will double check to confirm if some of these can't be moved there.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.

kfaraz commented on code in PR #18302:
URL: https://github.com/apache/druid/pull/18302#discussion_r2236501693


##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/DruidContainerResource.java:
##########
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import org.apache.druid.java.util.common.FileUtils;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.testing.DruidCommand;
+import org.apache.druid.testing.DruidContainer;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.TestcontainerResource;
+import org.testcontainers.containers.BindMode;
+import org.testcontainers.containers.wait.strategy.Wait;
+import org.testcontainers.utility.DockerImageName;
+
+import java.io.File;
+import java.nio.file.Files;
+import java.nio.file.attribute.PosixFilePermissions;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+
+/**
+ * {@link TestcontainerResource} to run Druid services.
+ * Currently, only core extensions can be used out-of-the-box with these containers
+ * such as {@code druid-s3-extensions} or {@code postgresql-metadata-storage},
+ * simply by adding them to {@code druid.extensions.loadList}.
+ * <p>
+ * {@link DruidContainers} should be used only for testing backward compatiblity
+ * or a Docker-specific feature. For all other testing needs, use plain old
+ * {@code EmbeddedDruidServer} as they are much faster, allow easy debugging and
+ * do not require downloading any images.
+ */
+public class DruidContainerResource extends TestcontainerResource<DruidContainer>
+{
+  /**
+   * Java system property to specify the name of the Docker test image.
+   */
+  public static final String PROPERTY_TEST_IMAGE = "druid.testing.docker.image";
+
+  private static final Logger log = new Logger(DruidContainerResource.class);
+
+  /**
+   * Forbidden server properties that may be used by EmbeddedDruidServers but
+   * interfere with the functioning of DruidContainer-based services.
+   */
+  private static final Set<String> FORBIDDEN_PROPERTIES = Set.of(
+      "druid.extensions.modulesForEmbeddedTests",
+      "druid.emitter"
+  );
+
+  /**
+   * A static incremental ID is used instead of a random number to ensure that
+   * tests are more deterministic and easier to debug.
+   */
+  private static final AtomicInteger SERVER_ID = new AtomicInteger(0);
+
+  private final String name;
+  private final DruidCommand command;
+  private final Map<String, String> properties = new HashMap<>();
+  private final List<Integer> taskPorts = new ArrayList<>();
+
+  private int servicePort;
+  private DockerImageName imageName;
+  private EmbeddedDruidCluster cluster;
+
+  private String containerDirectory;
+
+  private MountedDir indexerLogsDeepStorageDirectory;
+  private MountedDir serviceLogsDirectory;
+  private MountedDir segmentDeepStorageDirectory;
+
+  DruidContainerResource(DruidCommand command)
+  {
+    this.name = StringUtils.format(
+        "container_%s_%d",
+        command.getName(),
+        SERVER_ID.incrementAndGet()
+    );
+    this.command = command;
+
+    Integer[] exposedPorts = command.getExposedPorts();
+    servicePort = exposedPorts[0];
+    taskPorts.addAll(Arrays.asList(exposedPorts).subList(1, exposedPorts.length));
+  }
+
+  public DruidContainerResource withPort(int port)
+  {
+    this.servicePort = port;
+    return this;
+  }
+
+  /**
+   * Used to bind the task ports on middle managers.
+   */
+  public DruidContainerResource withTaskPorts(int startPort, int workerCapacity)
+  {
+    taskPorts.clear();
+    for (int i = 0; i < workerCapacity; ++i) {
+      taskPorts.add(startPort + i);
+    }
+    addProperty("druid.worker.capacity", String.valueOf(workerCapacity));
+    addProperty("druid.indexer.runner.startPort", String.valueOf(startPort));
+    addProperty("druid.indexer.runner.endPort", String.valueOf(startPort + workerCapacity));
+
+    return this;
+  }
+
+  public DruidContainerResource withImage(DockerImageName imageName)
+  {
+    this.imageName = imageName;
+    return this;
+  }
+
+  /**
+   * Uses the Docker test image specified by the system property
+   * {@link #PROPERTY_TEST_IMAGE} for this container.
+   */
+  public DruidContainerResource withTestImage()
+  {
+    String imageName = Objects.requireNonNull(
+        System.getProperty(PROPERTY_TEST_IMAGE),
+        StringUtils.format("System property[%s] is not set", PROPERTY_TEST_IMAGE)
+    );
+    return withImage(DockerImageName.parse(imageName));
+  }
+
+  public DruidContainerResource addProperty(String key, String value)
+  {
+    properties.put(key, value);
+    return this;
+  }
+
+  @Override
+  public void beforeStart(EmbeddedDruidCluster cluster)
+  {
+    this.cluster = cluster;
+
+    // Mount directories used by the entire cluster (including embedded servers)
+    this.segmentDeepStorageDirectory = new MountedDir(
+        "/tmp/druid/deep-store",
+        cluster.getTestFolder().getOrCreateFolder("deep-store").getAbsolutePath()
+    );
+    this.indexerLogsDeepStorageDirectory = new MountedDir(
+        "/tmp/druid/indexer-logs",
+        cluster.getTestFolder().getOrCreateFolder("indexer-logs").getAbsolutePath()
+    );
+
+    // Mount directories used by this container for easier debugging with service logs
+    this.containerDirectory = cluster.getTestFolder().getOrCreateFolder(name).getAbsolutePath();
+    this.serviceLogsDirectory = new MountedDir(
+        "/opt/druid/log",

Review Comment:
   This is log location used by the Druid Docker image by default.



##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/DruidContainerResource.java:
##########
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import org.apache.druid.java.util.common.FileUtils;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.testing.DruidCommand;
+import org.apache.druid.testing.DruidContainer;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.TestcontainerResource;
+import org.testcontainers.containers.BindMode;
+import org.testcontainers.containers.wait.strategy.Wait;
+import org.testcontainers.utility.DockerImageName;
+
+import java.io.File;
+import java.nio.file.Files;
+import java.nio.file.attribute.PosixFilePermissions;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+
+/**
+ * {@link TestcontainerResource} to run Druid services.
+ * Currently, only core extensions can be used out-of-the-box with these containers
+ * such as {@code druid-s3-extensions} or {@code postgresql-metadata-storage},
+ * simply by adding them to {@code druid.extensions.loadList}.
+ * <p>
+ * {@link DruidContainers} should be used only for testing backward compatiblity
+ * or a Docker-specific feature. For all other testing needs, use plain old
+ * {@code EmbeddedDruidServer} as they are much faster, allow easy debugging and
+ * do not require downloading any images.
+ */
+public class DruidContainerResource extends TestcontainerResource<DruidContainer>
+{
+  /**
+   * Java system property to specify the name of the Docker test image.
+   */
+  public static final String PROPERTY_TEST_IMAGE = "druid.testing.docker.image";
+
+  private static final Logger log = new Logger(DruidContainerResource.class);
+
+  /**
+   * Forbidden server properties that may be used by EmbeddedDruidServers but
+   * interfere with the functioning of DruidContainer-based services.
+   */
+  private static final Set<String> FORBIDDEN_PROPERTIES = Set.of(
+      "druid.extensions.modulesForEmbeddedTests",
+      "druid.emitter"
+  );
+
+  /**
+   * A static incremental ID is used instead of a random number to ensure that
+   * tests are more deterministic and easier to debug.
+   */
+  private static final AtomicInteger SERVER_ID = new AtomicInteger(0);
+
+  private final String name;
+  private final DruidCommand command;
+  private final Map<String, String> properties = new HashMap<>();
+  private final List<Integer> taskPorts = new ArrayList<>();
+
+  private int servicePort;
+  private DockerImageName imageName;
+  private EmbeddedDruidCluster cluster;
+
+  private String containerDirectory;
+
+  private MountedDir indexerLogsDeepStorageDirectory;
+  private MountedDir serviceLogsDirectory;
+  private MountedDir segmentDeepStorageDirectory;
+
+  DruidContainerResource(DruidCommand command)
+  {
+    this.name = StringUtils.format(
+        "container_%s_%d",
+        command.getName(),
+        SERVER_ID.incrementAndGet()
+    );
+    this.command = command;
+
+    Integer[] exposedPorts = command.getExposedPorts();
+    servicePort = exposedPorts[0];
+    taskPorts.addAll(Arrays.asList(exposedPorts).subList(1, exposedPorts.length));
+  }
+
+  public DruidContainerResource withPort(int port)
+  {
+    this.servicePort = port;
+    return this;
+  }
+
+  /**
+   * Used to bind the task ports on middle managers.
+   */
+  public DruidContainerResource withTaskPorts(int startPort, int workerCapacity)
+  {
+    taskPorts.clear();
+    for (int i = 0; i < workerCapacity; ++i) {
+      taskPorts.add(startPort + i);
+    }
+    addProperty("druid.worker.capacity", String.valueOf(workerCapacity));
+    addProperty("druid.indexer.runner.startPort", String.valueOf(startPort));
+    addProperty("druid.indexer.runner.endPort", String.valueOf(startPort + workerCapacity));
+
+    return this;
+  }
+
+  public DruidContainerResource withImage(DockerImageName imageName)
+  {
+    this.imageName = imageName;
+    return this;
+  }
+
+  /**
+   * Uses the Docker test image specified by the system property
+   * {@link #PROPERTY_TEST_IMAGE} for this container.
+   */
+  public DruidContainerResource withTestImage()
+  {
+    String imageName = Objects.requireNonNull(
+        System.getProperty(PROPERTY_TEST_IMAGE),
+        StringUtils.format("System property[%s] is not set", PROPERTY_TEST_IMAGE)
+    );
+    return withImage(DockerImageName.parse(imageName));
+  }
+
+  public DruidContainerResource addProperty(String key, String value)
+  {
+    properties.put(key, value);
+    return this;
+  }
+
+  @Override
+  public void beforeStart(EmbeddedDruidCluster cluster)
+  {
+    this.cluster = cluster;
+
+    // Mount directories used by the entire cluster (including embedded servers)
+    this.segmentDeepStorageDirectory = new MountedDir(
+        "/tmp/druid/deep-store",
+        cluster.getTestFolder().getOrCreateFolder("deep-store").getAbsolutePath()
+    );
+    this.indexerLogsDeepStorageDirectory = new MountedDir(
+        "/tmp/druid/indexer-logs",
+        cluster.getTestFolder().getOrCreateFolder("indexer-logs").getAbsolutePath()
+    );
+
+    // Mount directories used by this container for easier debugging with service logs
+    this.containerDirectory = cluster.getTestFolder().getOrCreateFolder(name).getAbsolutePath();
+    this.serviceLogsDirectory = new MountedDir(
+        "/opt/druid/log",

Review Comment:
   This is the location for log files used by the Druid Docker image by default.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.

kfaraz commented on code in PR #18302:
URL: https://github.com/apache/druid/pull/18302#discussion_r2236067684


##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/DruidContainerResource.java:
##########
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import org.apache.druid.java.util.common.FileUtils;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.testing.DruidCommand;
+import org.apache.druid.testing.DruidContainer;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.TestcontainerResource;
+import org.testcontainers.containers.BindMode;
+import org.testcontainers.containers.wait.strategy.Wait;
+import org.testcontainers.utility.DockerImageName;
+
+import java.io.File;
+import java.nio.file.Files;
+import java.nio.file.attribute.PosixFilePermissions;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+
+/**
+ * {@link TestcontainerResource} to run Druid services.
+ * Currently, only core extensions can be used out-of-the-box with these containers
+ * such as {@code druid-s3-extensions} or {@code postgresql-metadata-storage},
+ * simply by adding them to {@code druid.extensions.loadList}.
+ * <p>
+ * {@link DruidContainers} should be used only for testing backward compatiblity
+ * or a Docker-specific feature. For all other testing needs, use plain old
+ * {@code EmbeddedDruidServer} as they are much faster, allow easy debugging and
+ * do not require downloading any images.
+ */
+public class DruidContainerResource extends TestcontainerResource<DruidContainer>
+{
+  /**
+   * Java system property to specify the name of the Docker test image.
+   */
+  public static final String PROPERTY_TEST_IMAGE = "druid.testing.docker.image";
+
+  private static final Logger log = new Logger(DruidContainerResource.class);
+
+  /**
+   * Forbidden server properties that may be used by EmbeddedDruidServers but
+   * interfere with the functioning of DruidContainer-based services.
+   */
+  private static final Set<String> FORBIDDEN_PROPERTIES = Set.of(
+      "druid.extensions.modulesForEmbeddedTests",
+      "druid.emitter"
+  );
+
+  /**
+   * A static incremental ID is used instead of a random number to ensure that
+   * tests are more deterministic and easier to debug.
+   */
+  private static final AtomicInteger SERVER_ID = new AtomicInteger(0);
+
+  private final String name;
+  private final DruidCommand command;
+  private final Map<String, String> properties = new HashMap<>();
+  private final List<Integer> taskPorts = new ArrayList<>();
+
+  private int servicePort;
+  private DockerImageName imageName;
+  private EmbeddedDruidCluster cluster;
+
+  private String containerDirectory;
+
+  private MountedDir indexerLogsDeepStorageDirectory;
+  private MountedDir serviceLogsDirectory;
+  private MountedDir segmentDeepStorageDirectory;
+
+  DruidContainerResource(DruidCommand command)
+  {
+    this.name = StringUtils.format(
+        "container_%s_%d",
+        command.getName(),
+        SERVER_ID.incrementAndGet()
+    );
+    this.command = command;
+
+    Integer[] exposedPorts = command.getExposedPorts();
+    servicePort = exposedPorts[0];
+    taskPorts.addAll(Arrays.asList(exposedPorts).subList(1, exposedPorts.length));
+  }
+
+  public DruidContainerResource withPort(int port)
+  {
+    this.servicePort = port;
+    return this;
+  }
+
+  /**
+   * Used to bind the task ports on middle managers.
+   */
+  public DruidContainerResource withTaskPorts(int startPort, int workerCapacity)
+  {
+    taskPorts.clear();
+    for (int i = 0; i < workerCapacity; ++i) {
+      taskPorts.add(startPort + i);
+    }
+    addProperty("druid.worker.capacity", String.valueOf(workerCapacity));
+    addProperty("druid.indexer.runner.startPort", String.valueOf(startPort));
+    addProperty("druid.indexer.runner.endPort", String.valueOf(startPort + workerCapacity));
+
+    return this;
+  }
+
+  public DruidContainerResource withImage(DockerImageName imageName)
+  {
+    this.imageName = imageName;
+    return this;
+  }
+
+  /**
+   * Uses the Docker test image specified by the system property
+   * {@link #PROPERTY_TEST_IMAGE} for this container.
+   */
+  public DruidContainerResource withTestImage()
+  {
+    String imageName = Objects.requireNonNull(
+        System.getProperty(PROPERTY_TEST_IMAGE),
+        StringUtils.format("System property[%s] is not set", PROPERTY_TEST_IMAGE)
+    );
+    return withImage(DockerImageName.parse(imageName));
+  }
+
+  public DruidContainerResource addProperty(String key, String value)
+  {
+    properties.put(key, value);
+    return this;
+  }
+
+  @Override
+  public void beforeStart(EmbeddedDruidCluster cluster)
+  {
+    this.cluster = cluster;
+
+    // Mount directories used by the entire cluster (including embedded servers)
+    this.segmentDeepStorageDirectory = new MountedDir(
+        "/tmp/druid/deep-store",
+        cluster.getTestFolder().getOrCreateFolder("deep-store").getAbsolutePath()
+    );
+    this.indexerLogsDeepStorageDirectory = new MountedDir(
+        "/tmp/druid/indexer-logs",
+        cluster.getTestFolder().getOrCreateFolder("indexer-logs").getAbsolutePath()
+    );
+
+    // Mount directories used by this container for easier debugging with service logs
+    this.containerDirectory = cluster.getTestFolder().getOrCreateFolder(name).getAbsolutePath();
+    this.serviceLogsDirectory = new MountedDir(
+        "/opt/druid/log",
+        containerDirectory + "/log"
+    );
+
+    // Create the log directory upfront to avoid permission issues
+    createLogDirectory(new File(containerDirectory, "log"));
+  }
+
+  @Override
+  protected DruidContainer createContainer()
+  {
+    final List<Integer> exposedPorts = new ArrayList<>(taskPorts);
+    exposedPorts.add(servicePort);
+    log.info(
+        "Starting Druid container[%s] on port[%d] with mounted directory[%s] and exposed ports[%s].",
+        name, servicePort, containerDirectory, exposedPorts
+    );
+
+    final DruidContainer container = new DruidContainer(command, imageName)
+        .withExposedPorts(exposedPorts.toArray(new Integer[0]))
+        .withCommonProperties(getCommonProperties())
+        .withServiceProperties(getServerProperties())
+        .withFileSystemBind(
+            segmentDeepStorageDirectory.hostPath,
+            segmentDeepStorageDirectory.containerPath,
+            BindMode.READ_WRITE
+        )
+        .withFileSystemBind(
+            indexerLogsDeepStorageDirectory.hostPath,
+            indexerLogsDeepStorageDirectory.containerPath,
+            BindMode.READ_WRITE
+        )
+        .withFileSystemBind(
+            serviceLogsDirectory.hostPath,
+            serviceLogsDirectory.containerPath,
+            BindMode.READ_WRITE
+        )
+        .withEnv(
+            Map.of(
+                "DRUID_SET_HOST_IP", "0",
+                "DRUID_SET_HOST", "0"
+            )
+        )
+        .waitingFor(Wait.forHttp("/status/health").forPort(servicePort));
+
+    // Bind the ports statically (rather than using a mapped port) to the same
+    // value used in `druid.plaintextPort` to make this node discoverable
+    // by other services (both embedded and dockerized).
+    List<String> portBindings = exposedPorts.stream().map(
+        port -> StringUtils.format("%d:%d", port, port)
+    ).collect(Collectors.toList());
+    container.setPortBindings(portBindings);

Review Comment:
   This is a method exposed by `GenericContainer` itself, not added by this PR.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.

kfaraz commented on code in PR #18302:
URL: https://github.com/apache/druid/pull/18302#discussion_r2236075239


##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/DruidContainerResource.java:
##########
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import org.apache.druid.java.util.common.FileUtils;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.testing.DruidCommand;
+import org.apache.druid.testing.DruidContainer;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.TestcontainerResource;
+import org.testcontainers.containers.BindMode;
+import org.testcontainers.containers.wait.strategy.Wait;
+import org.testcontainers.utility.DockerImageName;
+
+import java.io.File;
+import java.nio.file.Files;
+import java.nio.file.attribute.PosixFilePermissions;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+
+/**
+ * {@link TestcontainerResource} to run Druid services.
+ * Currently, only core extensions can be used out-of-the-box with these containers
+ * such as {@code druid-s3-extensions} or {@code postgresql-metadata-storage},
+ * simply by adding them to {@code druid.extensions.loadList}.
+ * <p>
+ * {@link DruidContainers} should be used only for testing backward compatiblity
+ * or a Docker-specific feature. For all other testing needs, use plain old
+ * {@code EmbeddedDruidServer} as they are much faster, allow easy debugging and
+ * do not require downloading any images.
+ */
+public class DruidContainerResource extends TestcontainerResource<DruidContainer>
+{
+  /**
+   * Java system property to specify the name of the Docker test image.
+   */
+  public static final String PROPERTY_TEST_IMAGE = "druid.testing.docker.image";
+
+  private static final Logger log = new Logger(DruidContainerResource.class);
+
+  /**
+   * Forbidden server properties that may be used by EmbeddedDruidServers but
+   * interfere with the functioning of DruidContainer-based services.
+   */
+  private static final Set<String> FORBIDDEN_PROPERTIES = Set.of(
+      "druid.extensions.modulesForEmbeddedTests",
+      "druid.emitter"
+  );
+
+  /**
+   * A static incremental ID is used instead of a random number to ensure that
+   * tests are more deterministic and easier to debug.
+   */
+  private static final AtomicInteger SERVER_ID = new AtomicInteger(0);
+
+  private final String name;
+  private final DruidCommand command;
+  private final Map<String, String> properties = new HashMap<>();
+  private final List<Integer> taskPorts = new ArrayList<>();
+
+  private int servicePort;
+  private DockerImageName imageName;
+  private EmbeddedDruidCluster cluster;
+
+  private String containerDirectory;
+
+  private MountedDir indexerLogsDeepStorageDirectory;
+  private MountedDir serviceLogsDirectory;
+  private MountedDir segmentDeepStorageDirectory;
+
+  DruidContainerResource(DruidCommand command)
+  {
+    this.name = StringUtils.format(
+        "container_%s_%d",
+        command.getName(),
+        SERVER_ID.incrementAndGet()
+    );
+    this.command = command;
+
+    Integer[] exposedPorts = command.getExposedPorts();
+    servicePort = exposedPorts[0];
+    taskPorts.addAll(Arrays.asList(exposedPorts).subList(1, exposedPorts.length));
+  }
+
+  public DruidContainerResource withPort(int port)
+  {
+    this.servicePort = port;
+    return this;
+  }
+
+  /**
+   * Used to bind the task ports on middle managers.
+   */
+  public DruidContainerResource withTaskPorts(int startPort, int workerCapacity)
+  {
+    taskPorts.clear();
+    for (int i = 0; i < workerCapacity; ++i) {
+      taskPorts.add(startPort + i);
+    }
+    addProperty("druid.worker.capacity", String.valueOf(workerCapacity));
+    addProperty("druid.indexer.runner.startPort", String.valueOf(startPort));
+    addProperty("druid.indexer.runner.endPort", String.valueOf(startPort + workerCapacity));
+
+    return this;
+  }
+
+  public DruidContainerResource withImage(DockerImageName imageName)
+  {
+    this.imageName = imageName;
+    return this;
+  }
+
+  /**
+   * Uses the Docker test image specified by the system property
+   * {@link #PROPERTY_TEST_IMAGE} for this container.
+   */
+  public DruidContainerResource withTestImage()
+  {
+    String imageName = Objects.requireNonNull(
+        System.getProperty(PROPERTY_TEST_IMAGE),
+        StringUtils.format("System property[%s] is not set", PROPERTY_TEST_IMAGE)
+    );
+    return withImage(DockerImageName.parse(imageName));
+  }
+
+  public DruidContainerResource addProperty(String key, String value)
+  {
+    properties.put(key, value);
+    return this;
+  }
+
+  @Override
+  public void beforeStart(EmbeddedDruidCluster cluster)
+  {
+    this.cluster = cluster;
+
+    // Mount directories used by the entire cluster (including embedded servers)
+    this.segmentDeepStorageDirectory = new MountedDir(
+        "/tmp/druid/deep-store",
+        cluster.getTestFolder().getOrCreateFolder("deep-store").getAbsolutePath()
+    );
+    this.indexerLogsDeepStorageDirectory = new MountedDir(
+        "/tmp/druid/indexer-logs",
+        cluster.getTestFolder().getOrCreateFolder("indexer-logs").getAbsolutePath()
+    );
+
+    // Mount directories used by this container for easier debugging with service logs
+    this.containerDirectory = cluster.getTestFolder().getOrCreateFolder(name).getAbsolutePath();
+    this.serviceLogsDirectory = new MountedDir(
+        "/opt/druid/log",
+        containerDirectory + "/log"
+    );
+
+    // Create the log directory upfront to avoid permission issues
+    createLogDirectory(new File(containerDirectory, "log"));
+  }
+
+  @Override
+  protected DruidContainer createContainer()
+  {
+    final List<Integer> exposedPorts = new ArrayList<>(taskPorts);
+    exposedPorts.add(servicePort);
+    log.info(
+        "Starting Druid container[%s] on port[%d] with mounted directory[%s] and exposed ports[%s].",
+        name, servicePort, containerDirectory, exposedPorts
+    );
+
+    final DruidContainer container = new DruidContainer(command, imageName)
+        .withExposedPorts(exposedPorts.toArray(new Integer[0]))
+        .withCommonProperties(getCommonProperties())
+        .withServiceProperties(getServerProperties())
+        .withFileSystemBind(
+            segmentDeepStorageDirectory.hostPath,
+            segmentDeepStorageDirectory.containerPath,
+            BindMode.READ_WRITE
+        )

Review Comment:
   Since the `MountedDir` class is only needed inside the `DruidContainerResource`, I didn't create it as a top-level class. It is used only to invoke `GenericContainer.withFileSystemBind`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.

kfaraz commented on code in PR #18302:
URL: https://github.com/apache/druid/pull/18302#discussion_r2236837699


##########
.github/workflows/docker-tests.yml:
##########
@@ -0,0 +1,97 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: "Docker Tests using Distribution Image"
+on:
+  workflow_call:
+
+jobs:
+  run-docker-tests:
+    name: Run Docker tests
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Retrieve Docker container
+        id: docker-restore
+        uses: actions/cache/restore@v4
+        with:
+          key: druid-dist-container.tar.gz-${{ github.sha }}
+          path: |
+            ./druid-dist-container.tar.gz
+      - name: Stop and remove Druid Docker containers
+        run: |
+          echo "Force stopping all Druid containers and pruning"
+          docker ps -aq --filter "ancestor=apache/druid" | xargs -r docker rm -f
+          docker system prune -af --volumes
+      - name: Load Docker image
+        run: |
+          docker load --input druid-dist-container.tar.gz
+          docker images
+      - name: Setup Java
+        uses: actions/setup-java@v4
+        with:
+          distribution: 'zulu'
+          java-version: 17
+          cache: 'maven'
+      - name: Run *DockerTest*
+        id: run-it
+        run: |
+          source .github/scripts/distribution_checks_env.sh
+          .github/scripts/run_unit-tests -Dtest=*DockerTest* -Ddruid.testing.docker.image=$DRUID_DIST_IMAGE_NAME

Review Comment:
   > instead of trying to reuse the run_unit-tests script; you could possibly make a separate one for this; and run that (you could still invode run_unit-tests from there if you want)
   
   Could you please elaborate? What would be the advantage of putting this in a separate script if we would still invoke `run_unit-tests` from there?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "kgyrtkirk (via GitHub)" <gi...@apache.org>.

kgyrtkirk commented on code in PR #18302:
URL: https://github.com/apache/druid/pull/18302#discussion_r2237230677


##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/DruidContainerResource.java:
##########
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import org.apache.druid.java.util.common.FileUtils;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.testing.DruidCommand;
+import org.apache.druid.testing.DruidContainer;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.TestcontainerResource;
+import org.testcontainers.containers.BindMode;
+import org.testcontainers.containers.wait.strategy.Wait;
+import org.testcontainers.utility.DockerImageName;
+
+import java.io.File;
+import java.nio.file.Files;
+import java.nio.file.attribute.PosixFilePermissions;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+
+/**
+ * {@link TestcontainerResource} to run Druid services.
+ * Currently, only core extensions can be used out-of-the-box with these containers
+ * such as {@code druid-s3-extensions} or {@code postgresql-metadata-storage},
+ * simply by adding them to {@code druid.extensions.loadList}.
+ * <p>
+ * {@link DruidContainers} should be used only for testing backward compatiblity
+ * or a Docker-specific feature. For all other testing needs, use plain old
+ * {@code EmbeddedDruidServer} as they are much faster, allow easy debugging and
+ * do not require downloading any images.
+ */
+public class DruidContainerResource extends TestcontainerResource<DruidContainer>
+{
+  /**
+   * Java system property to specify the name of the Docker test image.
+   */
+  public static final String PROPERTY_TEST_IMAGE = "druid.testing.docker.image";
+
+  private static final Logger log = new Logger(DruidContainerResource.class);
+
+  /**
+   * Forbidden server properties that may be used by EmbeddedDruidServers but
+   * interfere with the functioning of DruidContainer-based services.
+   */
+  private static final Set<String> FORBIDDEN_PROPERTIES = Set.of(
+      "druid.extensions.modulesForEmbeddedTests",
+      "druid.emitter"
+  );
+
+  /**
+   * A static incremental ID is used instead of a random number to ensure that
+   * tests are more deterministic and easier to debug.
+   */
+  private static final AtomicInteger SERVER_ID = new AtomicInteger(0);
+
+  private final String name;
+  private final DruidCommand command;
+  private final Map<String, String> properties = new HashMap<>();
+  private final List<Integer> taskPorts = new ArrayList<>();
+
+  private int servicePort;
+  private DockerImageName imageName;
+  private EmbeddedDruidCluster cluster;
+
+  private String containerDirectory;
+
+  private MountedDir indexerLogsDeepStorageDirectory;
+  private MountedDir serviceLogsDirectory;
+  private MountedDir segmentDeepStorageDirectory;
+
+  DruidContainerResource(DruidCommand command)
+  {
+    this.name = StringUtils.format(
+        "container_%s_%d",
+        command.getName(),
+        SERVER_ID.incrementAndGet()
+    );
+    this.command = command;
+
+    Integer[] exposedPorts = command.getExposedPorts();
+    servicePort = exposedPorts[0];
+    taskPorts.addAll(Arrays.asList(exposedPorts).subList(1, exposedPorts.length));
+  }
+
+  public DruidContainerResource withPort(int port)
+  {
+    this.servicePort = port;
+    return this;
+  }
+
+  /**
+   * Used to bind the task ports on middle managers.
+   */
+  public DruidContainerResource withTaskPorts(int startPort, int workerCapacity)
+  {
+    taskPorts.clear();
+    for (int i = 0; i < workerCapacity; ++i) {
+      taskPorts.add(startPort + i);
+    }
+    addProperty("druid.worker.capacity", String.valueOf(workerCapacity));
+    addProperty("druid.indexer.runner.startPort", String.valueOf(startPort));
+    addProperty("druid.indexer.runner.endPort", String.valueOf(startPort + workerCapacity));
+
+    return this;
+  }
+
+  public DruidContainerResource withImage(DockerImageName imageName)
+  {
+    this.imageName = imageName;
+    return this;
+  }
+
+  /**
+   * Uses the Docker test image specified by the system property
+   * {@link #PROPERTY_TEST_IMAGE} for this container.
+   */
+  public DruidContainerResource withTestImage()
+  {
+    String imageName = Objects.requireNonNull(
+        System.getProperty(PROPERTY_TEST_IMAGE),
+        StringUtils.format("System property[%s] is not set", PROPERTY_TEST_IMAGE)
+    );
+    return withImage(DockerImageName.parse(imageName));
+  }
+
+  public DruidContainerResource addProperty(String key, String value)
+  {
+    properties.put(key, value);
+    return this;
+  }
+
+  @Override
+  public void beforeStart(EmbeddedDruidCluster cluster)
+  {
+    this.cluster = cluster;
+
+    // Mount directories used by the entire cluster (including embedded servers)
+    this.segmentDeepStorageDirectory = new MountedDir(
+        "/tmp/druid/deep-store",
+        cluster.getTestFolder().getOrCreateFolder("deep-store").getAbsolutePath()
+    );
+    this.indexerLogsDeepStorageDirectory = new MountedDir(
+        "/tmp/druid/indexer-logs",
+        cluster.getTestFolder().getOrCreateFolder("indexer-logs").getAbsolutePath()
+    );
+
+    // Mount directories used by this container for easier debugging with service logs
+    this.containerDirectory = cluster.getTestFolder().getOrCreateFolder(name).getAbsolutePath();
+    this.serviceLogsDirectory = new MountedDir(
+        "/opt/druid/log",
+        containerDirectory + "/log"
+    );
+
+    // Create the log directory upfront to avoid permission issues
+    createLogDirectory(new File(containerDirectory, "log"));
+  }
+
+  @Override
+  protected DruidContainer createContainer()
+  {
+    final List<Integer> exposedPorts = new ArrayList<>(taskPorts);
+    exposedPorts.add(servicePort);
+    log.info(
+        "Starting Druid container[%s] on port[%d] with mounted directory[%s] and exposed ports[%s].",
+        name, servicePort, containerDirectory, exposedPorts
+    );
+
+    final DruidContainer container = new DruidContainer(command, imageName)
+        .withExposedPorts(exposedPorts.toArray(new Integer[0]))
+        .withCommonProperties(getCommonProperties())
+        .withServiceProperties(getServerProperties())
+        .withFileSystemBind(
+            segmentDeepStorageDirectory.hostPath,
+            segmentDeepStorageDirectory.containerPath,
+            BindMode.READ_WRITE
+        )

Review Comment:
   yes - it is used for that ; but `MountedDir` seemed like a concept you added as it was usefull...
   usage of those concepts could be promoted with methods/etc I think that makes it easier to work with it....



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.

kfaraz commented on code in PR #18302:
URL: https://github.com/apache/druid/pull/18302#discussion_r2237236219


##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/DruidContainerResource.java:
##########
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import org.apache.druid.java.util.common.FileUtils;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.testing.DruidCommand;
+import org.apache.druid.testing.DruidContainer;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.TestcontainerResource;
+import org.testcontainers.containers.BindMode;
+import org.testcontainers.containers.wait.strategy.Wait;
+import org.testcontainers.utility.DockerImageName;
+
+import java.io.File;
+import java.nio.file.Files;
+import java.nio.file.attribute.PosixFilePermissions;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+
+/**
+ * {@link TestcontainerResource} to run Druid services.
+ * Currently, only core extensions can be used out-of-the-box with these containers
+ * such as {@code druid-s3-extensions} or {@code postgresql-metadata-storage},
+ * simply by adding them to {@code druid.extensions.loadList}.
+ * <p>
+ * {@link DruidContainers} should be used only for testing backward compatiblity
+ * or a Docker-specific feature. For all other testing needs, use plain old
+ * {@code EmbeddedDruidServer} as they are much faster, allow easy debugging and
+ * do not require downloading any images.
+ */
+public class DruidContainerResource extends TestcontainerResource<DruidContainer>
+{
+  /**
+   * Java system property to specify the name of the Docker test image.
+   */
+  public static final String PROPERTY_TEST_IMAGE = "druid.testing.docker.image";
+
+  private static final Logger log = new Logger(DruidContainerResource.class);
+
+  /**
+   * Forbidden server properties that may be used by EmbeddedDruidServers but
+   * interfere with the functioning of DruidContainer-based services.
+   */
+  private static final Set<String> FORBIDDEN_PROPERTIES = Set.of(
+      "druid.extensions.modulesForEmbeddedTests",
+      "druid.emitter"
+  );
+
+  /**
+   * A static incremental ID is used instead of a random number to ensure that
+   * tests are more deterministic and easier to debug.
+   */
+  private static final AtomicInteger SERVER_ID = new AtomicInteger(0);
+
+  private final String name;
+  private final DruidCommand command;
+  private final Map<String, String> properties = new HashMap<>();
+  private final List<Integer> taskPorts = new ArrayList<>();
+
+  private int servicePort;
+  private DockerImageName imageName;
+  private EmbeddedDruidCluster cluster;
+
+  private String containerDirectory;
+
+  private MountedDir indexerLogsDeepStorageDirectory;
+  private MountedDir serviceLogsDirectory;
+  private MountedDir segmentDeepStorageDirectory;
+
+  DruidContainerResource(DruidCommand command)
+  {
+    this.name = StringUtils.format(
+        "container_%s_%d",
+        command.getName(),
+        SERVER_ID.incrementAndGet()
+    );
+    this.command = command;
+
+    Integer[] exposedPorts = command.getExposedPorts();
+    servicePort = exposedPorts[0];
+    taskPorts.addAll(Arrays.asList(exposedPorts).subList(1, exposedPorts.length));
+  }
+
+  public DruidContainerResource withPort(int port)
+  {
+    this.servicePort = port;
+    return this;
+  }
+
+  /**
+   * Used to bind the task ports on middle managers.
+   */
+  public DruidContainerResource withTaskPorts(int startPort, int workerCapacity)
+  {
+    taskPorts.clear();
+    for (int i = 0; i < workerCapacity; ++i) {
+      taskPorts.add(startPort + i);
+    }
+    addProperty("druid.worker.capacity", String.valueOf(workerCapacity));
+    addProperty("druid.indexer.runner.startPort", String.valueOf(startPort));
+    addProperty("druid.indexer.runner.endPort", String.valueOf(startPort + workerCapacity));
+
+    return this;
+  }
+
+  public DruidContainerResource withImage(DockerImageName imageName)
+  {
+    this.imageName = imageName;
+    return this;
+  }
+
+  /**
+   * Uses the Docker test image specified by the system property
+   * {@link #PROPERTY_TEST_IMAGE} for this container.
+   */
+  public DruidContainerResource withTestImage()
+  {
+    String imageName = Objects.requireNonNull(
+        System.getProperty(PROPERTY_TEST_IMAGE),
+        StringUtils.format("System property[%s] is not set", PROPERTY_TEST_IMAGE)
+    );
+    return withImage(DockerImageName.parse(imageName));
+  }
+
+  public DruidContainerResource addProperty(String key, String value)
+  {
+    properties.put(key, value);
+    return this;
+  }
+
+  @Override
+  public void beforeStart(EmbeddedDruidCluster cluster)
+  {
+    this.cluster = cluster;
+
+    // Mount directories used by the entire cluster (including embedded servers)
+    this.segmentDeepStorageDirectory = new MountedDir(
+        "/tmp/druid/deep-store",
+        cluster.getTestFolder().getOrCreateFolder("deep-store").getAbsolutePath()
+    );
+    this.indexerLogsDeepStorageDirectory = new MountedDir(
+        "/tmp/druid/indexer-logs",
+        cluster.getTestFolder().getOrCreateFolder("indexer-logs").getAbsolutePath()
+    );
+
+    // Mount directories used by this container for easier debugging with service logs
+    this.containerDirectory = cluster.getTestFolder().getOrCreateFolder(name).getAbsolutePath();
+    this.serviceLogsDirectory = new MountedDir(
+        "/opt/druid/log",
+        containerDirectory + "/log"
+    );
+
+    // Create the log directory upfront to avoid permission issues
+    createLogDirectory(new File(containerDirectory, "log"));
+  }
+
+  @Override
+  protected DruidContainer createContainer()
+  {
+    final List<Integer> exposedPorts = new ArrayList<>(taskPorts);
+    exposedPorts.add(servicePort);
+    log.info(
+        "Starting Druid container[%s] on port[%d] with mounted directory[%s] and exposed ports[%s].",
+        name, servicePort, containerDirectory, exposedPorts
+    );
+
+    final DruidContainer container = new DruidContainer(command, imageName)
+        .withExposedPorts(exposedPorts.toArray(new Integer[0]))
+        .withCommonProperties(getCommonProperties())
+        .withServiceProperties(getServerProperties())
+        .withFileSystemBind(
+            segmentDeepStorageDirectory.hostPath,
+            segmentDeepStorageDirectory.containerPath,
+            BindMode.READ_WRITE
+        )

Review Comment:
   Fair enough. I guess we could add a method on `DruidContainer` which uses `MountedDir`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "kgyrtkirk (via GitHub)" <gi...@apache.org>.

kgyrtkirk commented on code in PR #18302:
URL: https://github.com/apache/druid/pull/18302#discussion_r2237225121


##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/DruidContainerResource.java:
##########
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import org.apache.druid.java.util.common.FileUtils;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.testing.DruidCommand;
+import org.apache.druid.testing.DruidContainer;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.TestcontainerResource;
+import org.testcontainers.containers.BindMode;
+import org.testcontainers.containers.wait.strategy.Wait;
+import org.testcontainers.utility.DockerImageName;
+
+import java.io.File;
+import java.nio.file.Files;
+import java.nio.file.attribute.PosixFilePermissions;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+
+/**
+ * {@link TestcontainerResource} to run Druid services.
+ * Currently, only core extensions can be used out-of-the-box with these containers
+ * such as {@code druid-s3-extensions} or {@code postgresql-metadata-storage},
+ * simply by adding them to {@code druid.extensions.loadList}.
+ * <p>
+ * {@link DruidContainers} should be used only for testing backward compatiblity
+ * or a Docker-specific feature. For all other testing needs, use plain old
+ * {@code EmbeddedDruidServer} as they are much faster, allow easy debugging and
+ * do not require downloading any images.
+ */
+public class DruidContainerResource extends TestcontainerResource<DruidContainer>
+{
+  /**
+   * Java system property to specify the name of the Docker test image.
+   */
+  public static final String PROPERTY_TEST_IMAGE = "druid.testing.docker.image";
+
+  private static final Logger log = new Logger(DruidContainerResource.class);
+
+  /**
+   * Forbidden server properties that may be used by EmbeddedDruidServers but
+   * interfere with the functioning of DruidContainer-based services.
+   */
+  private static final Set<String> FORBIDDEN_PROPERTIES = Set.of(
+      "druid.extensions.modulesForEmbeddedTests",
+      "druid.emitter"
+  );
+
+  /**
+   * A static incremental ID is used instead of a random number to ensure that
+   * tests are more deterministic and easier to debug.
+   */
+  private static final AtomicInteger SERVER_ID = new AtomicInteger(0);
+
+  private final String name;
+  private final DruidCommand command;
+  private final Map<String, String> properties = new HashMap<>();
+  private final List<Integer> taskPorts = new ArrayList<>();
+
+  private int servicePort;
+  private DockerImageName imageName;
+  private EmbeddedDruidCluster cluster;
+
+  private String containerDirectory;
+
+  private MountedDir indexerLogsDeepStorageDirectory;
+  private MountedDir serviceLogsDirectory;
+  private MountedDir segmentDeepStorageDirectory;
+
+  DruidContainerResource(DruidCommand command)
+  {
+    this.name = StringUtils.format(
+        "container_%s_%d",
+        command.getName(),
+        SERVER_ID.incrementAndGet()
+    );
+    this.command = command;
+
+    Integer[] exposedPorts = command.getExposedPorts();
+    servicePort = exposedPorts[0];
+    taskPorts.addAll(Arrays.asList(exposedPorts).subList(1, exposedPorts.length));
+  }
+
+  public DruidContainerResource withPort(int port)
+  {
+    this.servicePort = port;
+    return this;
+  }
+
+  /**
+   * Used to bind the task ports on middle managers.
+   */
+  public DruidContainerResource withTaskPorts(int startPort, int workerCapacity)
+  {
+    taskPorts.clear();
+    for (int i = 0; i < workerCapacity; ++i) {
+      taskPorts.add(startPort + i);
+    }
+    addProperty("druid.worker.capacity", String.valueOf(workerCapacity));
+    addProperty("druid.indexer.runner.startPort", String.valueOf(startPort));
+    addProperty("druid.indexer.runner.endPort", String.valueOf(startPort + workerCapacity));
+
+    return this;
+  }
+
+  public DruidContainerResource withImage(DockerImageName imageName)
+  {
+    this.imageName = imageName;
+    return this;
+  }
+
+  /**
+   * Uses the Docker test image specified by the system property
+   * {@link #PROPERTY_TEST_IMAGE} for this container.
+   */
+  public DruidContainerResource withTestImage()
+  {
+    String imageName = Objects.requireNonNull(
+        System.getProperty(PROPERTY_TEST_IMAGE),
+        StringUtils.format("System property[%s] is not set", PROPERTY_TEST_IMAGE)
+    );
+    return withImage(DockerImageName.parse(imageName));
+  }
+
+  public DruidContainerResource addProperty(String key, String value)
+  {
+    properties.put(key, value);
+    return this;
+  }
+
+  @Override
+  public void beforeStart(EmbeddedDruidCluster cluster)
+  {
+    this.cluster = cluster;
+
+    // Mount directories used by the entire cluster (including embedded servers)
+    this.segmentDeepStorageDirectory = new MountedDir(
+        "/tmp/druid/deep-store",
+        cluster.getTestFolder().getOrCreateFolder("deep-store").getAbsolutePath()
+    );
+    this.indexerLogsDeepStorageDirectory = new MountedDir(
+        "/tmp/druid/indexer-logs",
+        cluster.getTestFolder().getOrCreateFolder("indexer-logs").getAbsolutePath()
+    );
+
+    // Mount directories used by this container for easier debugging with service logs
+    this.containerDirectory = cluster.getTestFolder().getOrCreateFolder(name).getAbsolutePath();
+    this.serviceLogsDirectory = new MountedDir(
+        "/opt/druid/log",
+        containerDirectory + "/log"
+    );
+
+    // Create the log directory upfront to avoid permission issues
+    createLogDirectory(new File(containerDirectory, "log"));
+  }
+
+  @Override
+  protected DruidContainer createContainer()
+  {
+    final List<Integer> exposedPorts = new ArrayList<>(taskPorts);
+    exposedPorts.add(servicePort);
+    log.info(
+        "Starting Druid container[%s] on port[%d] with mounted directory[%s] and exposed ports[%s].",
+        name, servicePort, containerDirectory, exposedPorts
+    );
+
+    final DruidContainer container = new DruidContainer(command, imageName)
+        .withExposedPorts(exposedPorts.toArray(new Integer[0]))

Review Comment:
   yes - but you are extending that class; `DruidContainer extends GenericContainer<DruidContainer>`
   you could make it adapt better for our needs...so that its easier to read...no need to limit to `GenericContainer` api usage
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.

kfaraz commented on code in PR #18302:
URL: https://github.com/apache/druid/pull/18302#discussion_r2237415660


##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/DruidContainerResource.java:
##########
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import org.apache.druid.java.util.common.FileUtils;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.testing.DruidCommand;
+import org.apache.druid.testing.DruidContainer;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.TestcontainerResource;
+import org.testcontainers.containers.BindMode;
+import org.testcontainers.containers.wait.strategy.Wait;
+import org.testcontainers.utility.DockerImageName;
+
+import java.io.File;
+import java.nio.file.Files;
+import java.nio.file.attribute.PosixFilePermissions;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+
+/**
+ * {@link TestcontainerResource} to run Druid services.
+ * Currently, only core extensions can be used out-of-the-box with these containers
+ * such as {@code druid-s3-extensions} or {@code postgresql-metadata-storage},
+ * simply by adding them to {@code druid.extensions.loadList}.
+ * <p>
+ * {@link DruidContainers} should be used only for testing backward compatiblity
+ * or a Docker-specific feature. For all other testing needs, use plain old
+ * {@code EmbeddedDruidServer} as they are much faster, allow easy debugging and
+ * do not require downloading any images.
+ */
+public class DruidContainerResource extends TestcontainerResource<DruidContainer>
+{
+  /**
+   * Java system property to specify the name of the Docker test image.
+   */
+  public static final String PROPERTY_TEST_IMAGE = "druid.testing.docker.image";
+
+  private static final Logger log = new Logger(DruidContainerResource.class);
+
+  /**
+   * Forbidden server properties that may be used by EmbeddedDruidServers but
+   * interfere with the functioning of DruidContainer-based services.
+   */
+  private static final Set<String> FORBIDDEN_PROPERTIES = Set.of(
+      "druid.extensions.modulesForEmbeddedTests",
+      "druid.emitter"
+  );
+
+  /**
+   * A static incremental ID is used instead of a random number to ensure that
+   * tests are more deterministic and easier to debug.
+   */
+  private static final AtomicInteger SERVER_ID = new AtomicInteger(0);
+
+  private final String name;
+  private final DruidCommand command;
+  private final Map<String, String> properties = new HashMap<>();
+  private final List<Integer> taskPorts = new ArrayList<>();
+
+  private int servicePort;
+  private DockerImageName imageName;
+  private EmbeddedDruidCluster cluster;
+
+  private String containerDirectory;
+
+  private MountedDir indexerLogsDeepStorageDirectory;
+  private MountedDir serviceLogsDirectory;
+  private MountedDir segmentDeepStorageDirectory;
+
+  DruidContainerResource(DruidCommand command)
+  {
+    this.name = StringUtils.format(
+        "container_%s_%d",
+        command.getName(),
+        SERVER_ID.incrementAndGet()
+    );
+    this.command = command;
+
+    Integer[] exposedPorts = command.getExposedPorts();
+    servicePort = exposedPorts[0];
+    taskPorts.addAll(Arrays.asList(exposedPorts).subList(1, exposedPorts.length));
+  }
+
+  public DruidContainerResource withPort(int port)
+  {
+    this.servicePort = port;
+    return this;
+  }
+
+  /**
+   * Used to bind the task ports on middle managers.
+   */
+  public DruidContainerResource withTaskPorts(int startPort, int workerCapacity)
+  {
+    taskPorts.clear();
+    for (int i = 0; i < workerCapacity; ++i) {
+      taskPorts.add(startPort + i);
+    }
+    addProperty("druid.worker.capacity", String.valueOf(workerCapacity));
+    addProperty("druid.indexer.runner.startPort", String.valueOf(startPort));
+    addProperty("druid.indexer.runner.endPort", String.valueOf(startPort + workerCapacity));
+
+    return this;
+  }
+
+  public DruidContainerResource withImage(DockerImageName imageName)
+  {
+    this.imageName = imageName;
+    return this;
+  }
+
+  /**
+   * Uses the Docker test image specified by the system property
+   * {@link #PROPERTY_TEST_IMAGE} for this container.
+   */
+  public DruidContainerResource withTestImage()
+  {
+    String imageName = Objects.requireNonNull(
+        System.getProperty(PROPERTY_TEST_IMAGE),
+        StringUtils.format("System property[%s] is not set", PROPERTY_TEST_IMAGE)
+    );
+    return withImage(DockerImageName.parse(imageName));
+  }
+
+  public DruidContainerResource addProperty(String key, String value)
+  {
+    properties.put(key, value);
+    return this;
+  }
+
+  @Override
+  public void beforeStart(EmbeddedDruidCluster cluster)
+  {
+    this.cluster = cluster;
+
+    // Mount directories used by the entire cluster (including embedded servers)
+    this.segmentDeepStorageDirectory = new MountedDir(
+        "/tmp/druid/deep-store",
+        cluster.getTestFolder().getOrCreateFolder("deep-store").getAbsolutePath()
+    );
+    this.indexerLogsDeepStorageDirectory = new MountedDir(
+        "/tmp/druid/indexer-logs",
+        cluster.getTestFolder().getOrCreateFolder("indexer-logs").getAbsolutePath()
+    );
+
+    // Mount directories used by this container for easier debugging with service logs
+    this.containerDirectory = cluster.getTestFolder().getOrCreateFolder(name).getAbsolutePath();
+    this.serviceLogsDirectory = new MountedDir(
+        "/opt/druid/log",
+        containerDirectory + "/log"
+    );
+
+    // Create the log directory upfront to avoid permission issues
+    createLogDirectory(new File(containerDirectory, "log"));
+  }
+
+  @Override
+  protected DruidContainer createContainer()
+  {
+    final List<Integer> exposedPorts = new ArrayList<>(taskPorts);
+    exposedPorts.add(servicePort);
+    log.info(
+        "Starting Druid container[%s] on port[%d] with mounted directory[%s] and exposed ports[%s].",
+        name, servicePort, containerDirectory, exposedPorts
+    );
+
+    final DruidContainer container = new DruidContainer(command, imageName)
+        .withExposedPorts(exposedPorts.toArray(new Integer[0]))
+        .withCommonProperties(getCommonProperties())
+        .withServiceProperties(getServerProperties())
+        .withFileSystemBind(
+            segmentDeepStorageDirectory.hostPath,
+            segmentDeepStorageDirectory.containerPath,
+            BindMode.READ_WRITE
+        )
+        .withFileSystemBind(
+            indexerLogsDeepStorageDirectory.hostPath,
+            indexerLogsDeepStorageDirectory.containerPath,
+            BindMode.READ_WRITE
+        )
+        .withFileSystemBind(
+            serviceLogsDirectory.hostPath,
+            serviceLogsDirectory.containerPath,
+            BindMode.READ_WRITE
+        )
+        .withEnv(
+            Map.of(
+                "DRUID_SET_HOST_IP", "0",
+                "DRUID_SET_HOST", "0"
+            )
+        )
+        .waitingFor(Wait.forHttp("/status/health").forPort(servicePort));
+
+    // Bind the ports statically (rather than using a mapped port) to the same
+    // value used in `druid.plaintextPort` to make this node discoverable
+    // by other services (both embedded and dockerized).
+    List<String> portBindings = exposedPorts.stream().map(
+        port -> StringUtils.format("%d:%d", port, port)
+    ).collect(Collectors.toList());
+    container.setPortBindings(portBindings);
+
+    return container;
+  }
+
+  /**
+   * Writes the common properties to a file.
+   *
+   * @return Name of the file.
+   */
+  private Properties getCommonProperties()
+  {
+    final Properties commonProperties = new Properties();
+    commonProperties.putAll(cluster.getCommonProperties());
+    FORBIDDEN_PROPERTIES.forEach(commonProperties::remove);
+
+    commonProperties.setProperty(
+        "druid.storage.storageDirectory",
+        segmentDeepStorageDirectory.containerPath
+    );
+    commonProperties.setProperty(
+        "druid.indexer.logs.directory",
+        indexerLogsDeepStorageDirectory.containerPath
+    );
+
+    log.info(
+        "Writing common properties for Druid container[%s]: [%s]",
+        name, commonProperties
+    );
+
+    return commonProperties;
+  }
+
+  /**
+   * Writes the server properties to a file.
+   *
+   * @return Name of the file.
+   */
+  private Properties getServerProperties()
+  {
+    FORBIDDEN_PROPERTIES.forEach(properties::remove);
+    addProperty("druid.host", EmbeddedDruidCluster.getDefaultHost());
+    addProperty("druid.plaintextPort", String.valueOf(servicePort));
+
+    final Properties serverProperties = new Properties();
+    serverProperties.putAll(properties);
+
+    log.info(
+        "Writing runtime properties for Druid container[%s]: [%s]",
+        name, serverProperties
+    );
+
+    return serverProperties;
+  }
+
+  private static void createLogDirectory(File dir)
+  {
+    try {
+      FileUtils.mkdirp(dir);
+      Files.setPosixFilePermissions(dir.toPath(), PosixFilePermissions.fromString("rwxrwxrwx"));
+    }
+    catch (Exception e) {
+      throw new RuntimeException(e);
+    }
+  }

Review Comment:
   Updated ,thanks for the suggestion!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "Akshat-Jain (via GitHub)" <gi...@apache.org>.

Akshat-Jain closed pull request #18302: Add DruidContainers to run docker tests with embedded-test framework
URL: https://github.com/apache/druid/pull/18302


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.

kfaraz commented on code in PR #18302:
URL: https://github.com/apache/druid/pull/18302#discussion_r2259594279


##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/IngestionBackwardCompatibilityDockerTest.java:
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.druid.client.coordinator.CoordinatorServiceClient;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.http.client.response.BytesFullResponseHandler;
+import org.apache.druid.java.util.http.client.response.BytesFullResponseHolder;
+import org.apache.druid.rpc.RequestBuilder;
+import org.apache.druid.rpc.indexing.SegmentUpdateResponse;
+import org.apache.druid.testing.DruidContainer;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.jboss.netty.handler.codec.http.HttpMethod;
+import org.junit.jupiter.api.Nested;
+
+/**
+ * Runs some basic ingestion tests using {@code DruidContainers} to verify
+ * backward compatibility with old Druid images.
+ */
+public class IngestionBackwardCompatibilityDockerTest
+{
+  @Nested
+  public class Apache31 extends IngestionDockerTest
+  {
+    @Override
+    public EmbeddedDruidCluster createCluster()
+    {
+      coordinator.usingImage(DruidContainer.Image.APACHE_31);

Review Comment:
   > Btw, IMO it's nicer to use annotations rather than test names to control what tests run in a standard mvn run. The downside of using names is that it's indirect. It makes it less obvious which tests run and which don't, and also means that if you want to change whether a test runs, you have to rename it.
   
   I agree. In the current code, I am using the `@Tag("docker-tests")` annotation to skip the Docker-based tests.
   The `*DockerTest*` class name is to tell `mvn verify` to look for these classes while searching for eligible integration tests. Otherwise, it only looks for `*IT` or `IT*` classes by default.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.

kfaraz commented on code in PR #18302:
URL: https://github.com/apache/druid/pull/18302#discussion_r2259611508


##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/IngestionBackwardCompatibilityDockerTest.java:
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.druid.client.coordinator.CoordinatorServiceClient;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.http.client.response.BytesFullResponseHandler;
+import org.apache.druid.java.util.http.client.response.BytesFullResponseHolder;
+import org.apache.druid.rpc.RequestBuilder;
+import org.apache.druid.rpc.indexing.SegmentUpdateResponse;
+import org.apache.druid.testing.DruidContainer;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.jboss.netty.handler.codec.http.HttpMethod;
+import org.junit.jupiter.api.Nested;
+
+/**
+ * Runs some basic ingestion tests using {@code DruidContainers} to verify
+ * backward compatibility with old Druid images.
+ */
+public class IngestionBackwardCompatibilityDockerTest
+{
+  @Nested
+  public class Apache31 extends IngestionDockerTest
+  {
+    @Override
+    public EmbeddedDruidCluster createCluster()
+    {
+      coordinator.usingImage(DruidContainer.Image.APACHE_31);

Review Comment:
   > But, the backwards compatibility test isn't intended to deploy the "current code". It wants to deploy a specific older version. So, it doesn't have that problem, and it's therefore good to always run it.
   
   Thanks for the suggestion!
   
   In that case, I will remove the `BackwardCompatiblityIngestionDockerTest` from the `DockerTestBase` hierarchy. I will also update it to use only Druid 31 containers and embedded servers for it (currently it uses a mix of Druid 31 containers, current code Druid containers and embedded servers).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.

kfaraz commented on PR #18302:
URL: https://github.com/apache/druid/pull/18302#issuecomment-3157295116

   Thanks for the suggestions, @capistrant ! I have updated the javadocs as suggested and replied to your other comments.
   Please let me know what you think.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.

kfaraz commented on code in PR #18302:
URL: https://github.com/apache/druid/pull/18302#discussion_r2255783749


##########
services/src/test/java/org/apache/druid/testing/embedded/EmbeddedDruidCluster.java:
##########
@@ -232,13 +228,21 @@ public TestFolder getTestFolder()
   }
 
   /**
-   * The embedded Zookeeper server used by this cluster, if any.
-   *
-   * @throws NullPointerException if this cluster has no embedded zookeeper.
+   * Uses a container-friendly hostname for all embedded services, Druid as well
+   * as external.
+   */
+  public EmbeddedDruidCluster useContainerFriendlyHostname()

Review Comment:
   Fair point!
   
   All the existing embedded tests would still work if we always used the `containerFriendly` hostname. (since the Docker tests already use embedded services and they are able to connect to each other seamlessly)
   In fact, that's how it used to be (except we were using `InetAddress.getLocalHost().getCanonicalHostName()` instead of `InetAddress.getLocalHost().getHostAddress()`),
   but we changed it in #18228 as there were some apprehensions with using the canonical host name,
   ref https://github.com/apache/druid/pull/18228/files#r2197340379.
   
   But now that we are using the `getHostAddress()` which is simply the IP address, I think we can just stick to using the `containerFriendly` one all the time.
   
   cc: @clintropolis 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.

kfaraz commented on code in PR #18302:
URL: https://github.com/apache/druid/pull/18302#discussion_r2259611508


##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/IngestionBackwardCompatibilityDockerTest.java:
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.druid.client.coordinator.CoordinatorServiceClient;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.http.client.response.BytesFullResponseHandler;
+import org.apache.druid.java.util.http.client.response.BytesFullResponseHolder;
+import org.apache.druid.rpc.RequestBuilder;
+import org.apache.druid.rpc.indexing.SegmentUpdateResponse;
+import org.apache.druid.testing.DruidContainer;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.jboss.netty.handler.codec.http.HttpMethod;
+import org.junit.jupiter.api.Nested;
+
+/**
+ * Runs some basic ingestion tests using {@code DruidContainers} to verify
+ * backward compatibility with old Druid images.
+ */
+public class IngestionBackwardCompatibilityDockerTest
+{
+  @Nested
+  public class Apache31 extends IngestionDockerTest
+  {
+    @Override
+    public EmbeddedDruidCluster createCluster()
+    {
+      coordinator.usingImage(DruidContainer.Image.APACHE_31);

Review Comment:
   > But, the backwards compatibility test isn't intended to deploy the "current code". It wants to deploy a specific older version. So, it doesn't have that problem, and it's therefore good to always run it.
   
   Thanks for the suggestion!
   
   In that case, I will remove the skip annotation from the backward compatibility test and use only Druid 31 containers and embedded servers for it (currently it uses a mix of Druid 31 containers, current code Druid containers and embedded servers).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.

kfaraz commented on code in PR #18302:
URL: https://github.com/apache/druid/pull/18302#discussion_r2244409629


##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/IngestionDockerTest.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.druid.common.utils.IdUtils;
+import org.apache.druid.data.input.impl.CsvInputFormat;
+import org.apache.druid.data.input.impl.DimensionsSpec;
+import org.apache.druid.data.input.impl.TimestampSpec;
+import org.apache.druid.indexer.TaskState;
+import org.apache.druid.indexer.TaskStatusPlus;
+import org.apache.druid.indexing.kafka.KafkaIndexTaskModule;
+import org.apache.druid.indexing.kafka.simulate.KafkaResource;
+import org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorIOConfig;
+import org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorSpec;
+import org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorTuningConfig;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.supervisor.SupervisorStatus;
+import org.apache.druid.java.util.common.DateTimes;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.parsers.CloseableIterator;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.indexing.DataSchema;
+import org.apache.druid.testing.embedded.EmbeddedBroker;
+import org.apache.druid.testing.embedded.EmbeddedClusterApis;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.EmbeddedOverlord;
+import org.apache.druid.testing.embedded.EmbeddedRouter;
+import org.apache.druid.testing.embedded.derby.EmbeddedDerbyMetadataResource;
+import org.apache.druid.testing.embedded.indexing.Resources;
+import org.apache.druid.testing.embedded.junit5.EmbeddedClusterTestBase;
+import org.apache.druid.testing.embedded.minio.MinIOStorageResource;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.joda.time.DateTime;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ThreadLocalRandom;
+
+/**
+ * Runs some basic ingestion tests using {@link DruidContainers}.
+ */
+public class IngestionDockerTest extends EmbeddedClusterTestBase
+{
+  static {
+    System.setProperty("druid.testing.docker.image", "apache/druid:35.0.0-SNAPSHOT");

Review Comment:
   > The error message when the property isn't set should spell this out. It should mention that the property is typically set during CI, but isn't set outside CI because we don't know what image you want to use.
   
   Updated the message in `DruidContainerResource`.



##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/IngestionDockerTest.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.druid.common.utils.IdUtils;
+import org.apache.druid.data.input.impl.CsvInputFormat;
+import org.apache.druid.data.input.impl.DimensionsSpec;
+import org.apache.druid.data.input.impl.TimestampSpec;
+import org.apache.druid.indexer.TaskState;
+import org.apache.druid.indexer.TaskStatusPlus;
+import org.apache.druid.indexing.kafka.KafkaIndexTaskModule;
+import org.apache.druid.indexing.kafka.simulate.KafkaResource;
+import org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorIOConfig;
+import org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorSpec;
+import org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorTuningConfig;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.supervisor.SupervisorStatus;
+import org.apache.druid.java.util.common.DateTimes;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.parsers.CloseableIterator;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.indexing.DataSchema;
+import org.apache.druid.testing.embedded.EmbeddedBroker;
+import org.apache.druid.testing.embedded.EmbeddedClusterApis;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.EmbeddedOverlord;
+import org.apache.druid.testing.embedded.EmbeddedRouter;
+import org.apache.druid.testing.embedded.derby.EmbeddedDerbyMetadataResource;
+import org.apache.druid.testing.embedded.indexing.Resources;
+import org.apache.druid.testing.embedded.junit5.EmbeddedClusterTestBase;
+import org.apache.druid.testing.embedded.minio.MinIOStorageResource;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.joda.time.DateTime;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ThreadLocalRandom;
+
+/**
+ * Runs some basic ingestion tests using {@link DruidContainers}.
+ */
+public class IngestionDockerTest extends EmbeddedClusterTestBase
+{
+  static {
+    System.setProperty("druid.testing.docker.image", "apache/druid:35.0.0-SNAPSHOT");
+  }
+
+  // Druid Docker containers
+  protected final DruidContainerResource overlordLeader = DruidContainers.newOverlord().withTestImage();
+  protected final DruidContainerResource coordinator = DruidContainers.newCoordinator().withTestImage();
+  protected final DruidContainerResource historical = DruidContainers.newHistorical().withTestImage();
+  protected final DruidContainerResource broker1 = DruidContainers.newBroker().withTestImage();
+  protected final DruidContainerResource middleManager = DruidContainers
+      .newMiddleManager()
+      .withTestImage()
+      .addProperty("druid.segment.handoff.pollDuration", "PT0.1s");
+
+  // Follower EmbeddedOverlord to watch segment publish events
+  private final EmbeddedOverlord overlordFollower = new EmbeddedOverlord()
+      .addProperty("druid.plaintextPort", "7090")
+      .addProperty("druid.manager.segments.useIncrementalCache", "always")
+      .addProperty("druid.manager.segments.pollDuration", "PT0.1s");
+
+  // Additional EmbeddedBroker to wait for segments to become queryable
+  private final EmbeddedBroker broker2 = new EmbeddedBroker()
+      .addProperty("druid.plaintextPort", "7082");
+
+  private final KafkaResource kafkaServer = new KafkaResource();
+
+  @Override
+  public EmbeddedDruidCluster createCluster()
+  {
+    final EmbeddedDruidCluster cluster = EmbeddedDruidCluster.withZookeeper();
+
+    return cluster
+        .useDruidContainers()
+        .useLatchableEmitter()
+        // Needed for overlordFollower to recognize the KafkaSupervisor type
+        .addExtension(KafkaIndexTaskModule.class)
+        .addResource(new EmbeddedDerbyMetadataResource())
+        .addResource(new MinIOStorageResource())
+        .addResource(kafkaServer)
+        .addCommonProperty(
+            "druid.extensions.loadList",
+            "[\"druid-s3-extensions\", \"druid-kafka-indexing-service\", \"druid-multi-stage-query\"]"
+        )
+        .addResource(coordinator)
+        .addResource(overlordLeader)
+        .addResource(middleManager)
+        .addResource(historical)
+        .addResource(broker1)
+        .addServer(overlordFollower)
+        .addServer(broker2)
+        .addServer(new EmbeddedRouter());
+  }
+
+  @Test
+  public void test_runIndexTask_andKillData()
+  {
+

Review Comment:
   added



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "Akshat-Jain (via GitHub)" <gi...@apache.org>.

Akshat-Jain closed pull request #18302: Add DruidContainers to run docker tests with embedded-test framework
URL: https://github.com/apache/druid/pull/18302


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.

kfaraz commented on code in PR #18302:
URL: https://github.com/apache/druid/pull/18302#discussion_r2236983793


##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/DruidContainerResource.java:
##########
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import org.apache.druid.java.util.common.FileUtils;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.testing.DruidCommand;
+import org.apache.druid.testing.DruidContainer;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.TestcontainerResource;
+import org.testcontainers.containers.BindMode;
+import org.testcontainers.containers.wait.strategy.Wait;
+import org.testcontainers.utility.DockerImageName;
+
+import java.io.File;
+import java.nio.file.Files;
+import java.nio.file.attribute.PosixFilePermissions;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+
+/**
+ * {@link TestcontainerResource} to run Druid services.
+ * Currently, only core extensions can be used out-of-the-box with these containers
+ * such as {@code druid-s3-extensions} or {@code postgresql-metadata-storage},
+ * simply by adding them to {@code druid.extensions.loadList}.
+ * <p>
+ * {@link DruidContainers} should be used only for testing backward compatiblity
+ * or a Docker-specific feature. For all other testing needs, use plain old
+ * {@code EmbeddedDruidServer} as they are much faster, allow easy debugging and
+ * do not require downloading any images.
+ */
+public class DruidContainerResource extends TestcontainerResource<DruidContainer>
+{
+  /**
+   * Java system property to specify the name of the Docker test image.
+   */
+  public static final String PROPERTY_TEST_IMAGE = "druid.testing.docker.image";
+
+  private static final Logger log = new Logger(DruidContainerResource.class);
+
+  /**
+   * Forbidden server properties that may be used by EmbeddedDruidServers but
+   * interfere with the functioning of DruidContainer-based services.
+   */
+  private static final Set<String> FORBIDDEN_PROPERTIES = Set.of(
+      "druid.extensions.modulesForEmbeddedTests",
+      "druid.emitter"
+  );
+
+  /**
+   * A static incremental ID is used instead of a random number to ensure that
+   * tests are more deterministic and easier to debug.
+   */
+  private static final AtomicInteger SERVER_ID = new AtomicInteger(0);
+
+  private final String name;
+  private final DruidCommand command;
+  private final Map<String, String> properties = new HashMap<>();
+  private final List<Integer> taskPorts = new ArrayList<>();
+
+  private int servicePort;
+  private DockerImageName imageName;
+  private EmbeddedDruidCluster cluster;
+
+  private String containerDirectory;
+
+  private MountedDir indexerLogsDeepStorageDirectory;
+  private MountedDir serviceLogsDirectory;
+  private MountedDir segmentDeepStorageDirectory;
+
+  DruidContainerResource(DruidCommand command)
+  {
+    this.name = StringUtils.format(
+        "container_%s_%d",
+        command.getName(),
+        SERVER_ID.incrementAndGet()
+    );
+    this.command = command;
+
+    Integer[] exposedPorts = command.getExposedPorts();
+    servicePort = exposedPorts[0];
+    taskPorts.addAll(Arrays.asList(exposedPorts).subList(1, exposedPorts.length));
+  }
+
+  public DruidContainerResource withPort(int port)
+  {
+    this.servicePort = port;
+    return this;
+  }
+
+  /**
+   * Used to bind the task ports on middle managers.
+   */
+  public DruidContainerResource withTaskPorts(int startPort, int workerCapacity)
+  {
+    taskPorts.clear();
+    for (int i = 0; i < workerCapacity; ++i) {
+      taskPorts.add(startPort + i);
+    }
+    addProperty("druid.worker.capacity", String.valueOf(workerCapacity));
+    addProperty("druid.indexer.runner.startPort", String.valueOf(startPort));
+    addProperty("druid.indexer.runner.endPort", String.valueOf(startPort + workerCapacity));
+
+    return this;
+  }
+
+  public DruidContainerResource withImage(DockerImageName imageName)
+  {
+    this.imageName = imageName;
+    return this;
+  }
+
+  /**
+   * Uses the Docker test image specified by the system property
+   * {@link #PROPERTY_TEST_IMAGE} for this container.
+   */
+  public DruidContainerResource withTestImage()
+  {
+    String imageName = Objects.requireNonNull(
+        System.getProperty(PROPERTY_TEST_IMAGE),
+        StringUtils.format("System property[%s] is not set", PROPERTY_TEST_IMAGE)
+    );
+    return withImage(DockerImageName.parse(imageName));
+  }
+
+  public DruidContainerResource addProperty(String key, String value)
+  {
+    properties.put(key, value);
+    return this;
+  }
+
+  @Override
+  public void beforeStart(EmbeddedDruidCluster cluster)
+  {
+    this.cluster = cluster;
+
+    // Mount directories used by the entire cluster (including embedded servers)
+    this.segmentDeepStorageDirectory = new MountedDir(
+        "/tmp/druid/deep-store",

Review Comment:
   Used `/druid/deep-store` instead.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.

kfaraz commented on code in PR #18302:
URL: https://github.com/apache/druid/pull/18302#discussion_r2236988838


##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/DruidContainers.java:
##########
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import org.apache.druid.testing.DruidCommand;
+
+/**
+ * Factory for {@link DruidContainerResource} that can run specific services.
+ *
+ * @see #newOverlord()
+ * @see #newCoordinator()
+ */
+public final class DruidContainers
+{
+  private DruidContainers()
+  {
+    // no instantiation
+  }
+
+  /**
+   * Creates a new {@link DruidContainerResource} to run a Coordinator node.
+   */
+  public static DruidContainerResource newCoordinator()
+  {
+    return new DruidContainerResource(DruidCommand.COORDINATOR)
+        .addProperty("druid.coordinator.startDelay", "PT0.1S")
+        .addProperty("druid.coordinator.period", "PT0.5S")
+        .addProperty("druid.manager.segments.pollDuration", "PT0.1S");
+  }
+
+  /**
+   * Creates a new {@link DruidContainerResource} to run an Overlord node.
+   */
+  public static DruidContainerResource newOverlord()
+  {
+    // Keep a small sync timeout so that Peons and Indexers are not stuck
+    // handling a change request when Overlord has already shutdown
+    return new DruidContainerResource(DruidCommand.OVERLORD)
+        .addProperty("druid.indexer.storage.type", "metadata")
+        .addProperty("druid.indexer.queue.startDelay", "PT0S")
+        .addProperty("druid.indexer.queue.restartDelay", "PT0S")
+        .addProperty("druid.indexer.runner.syncRequestTimeout", "PT1S");
+  }
+
+  /**
+   * Creates a new {@link DruidContainerResource} to run an Indexer node.
+   */
+  public static DruidContainerResource newIndexer()
+  {
+    return new DruidContainerResource(DruidCommand.INDEXER)
+        .addProperty("druid.lookup.enableLookupSyncOnStartup", "false")
+        .addProperty("druid.processing.buffer.sizeBytes", "50MiB")
+        .addProperty("druid.processing.numMergeBuffers", "2")
+        .addProperty("druid.processing.numThreads", "5");
+  }

Review Comment:
   For now, I have moved all required properties to `DruidCommand` so that all users of `DruidContainer` can get those default properties out of the box.



##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/DruidContainers.java:
##########
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import org.apache.druid.testing.DruidCommand;
+
+/**
+ * Factory for {@link DruidContainerResource} that can run specific services.
+ *
+ * @see #newOverlord()
+ * @see #newCoordinator()
+ */
+public final class DruidContainers
+{
+  private DruidContainers()
+  {
+    // no instantiation
+  }
+
+  /**
+   * Creates a new {@link DruidContainerResource} to run a Coordinator node.
+   */
+  public static DruidContainerResource newCoordinator()
+  {
+    return new DruidContainerResource(DruidCommand.COORDINATOR)
+        .addProperty("druid.coordinator.startDelay", "PT0.1S")
+        .addProperty("druid.coordinator.period", "PT0.5S")
+        .addProperty("druid.manager.segments.pollDuration", "PT0.1S");
+  }
+
+  /**
+   * Creates a new {@link DruidContainerResource} to run an Overlord node.
+   */
+  public static DruidContainerResource newOverlord()
+  {
+    // Keep a small sync timeout so that Peons and Indexers are not stuck
+    // handling a change request when Overlord has already shutdown
+    return new DruidContainerResource(DruidCommand.OVERLORD)
+        .addProperty("druid.indexer.storage.type", "metadata")
+        .addProperty("druid.indexer.queue.startDelay", "PT0S")
+        .addProperty("druid.indexer.queue.restartDelay", "PT0S")
+        .addProperty("druid.indexer.runner.syncRequestTimeout", "PT1S");
+  }
+
+  /**
+   * Creates a new {@link DruidContainerResource} to run an Indexer node.
+   */
+  public static DruidContainerResource newIndexer()
+  {
+    return new DruidContainerResource(DruidCommand.INDEXER)
+        .addProperty("druid.lookup.enableLookupSyncOnStartup", "false")
+        .addProperty("druid.processing.buffer.sizeBytes", "50MiB")
+        .addProperty("druid.processing.numMergeBuffers", "2")
+        .addProperty("druid.processing.numThreads", "5");
+  }

Review Comment:
   Some of these properties might make it into DruidCommand itself if we feel that these are always needed out of the box whenever running a DruidContainer.
   
   I have just kept the minimal set in DruidCommand for now but will double check to confirm if some of these can't be moved there.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.

kfaraz commented on PR #18302:
URL: https://github.com/apache/druid/pull/18302#issuecomment-3127789784

   Thanks for the suggestions, @kgyrtkirk !
   
   I have made the following changes:
   - Moved all required properties of `DruidCommand` so that all users of `DruidContainer` can benefit from the default set of properties.
   - Removed `distribution-checks.yml` workflow and `distribution_checks_env.sh` as you suggested
   - Used `File` instead of `String` where applicable
   - Used path `/druid/deep-store` on container
   
   I am not very clear on how the usage of `worker.yml`/`run_unit-tests` script in the new workflow.
   I am open to suggestions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.

kfaraz commented on code in PR #18302:
URL: https://github.com/apache/druid/pull/18302#discussion_r2239049579


##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/IngestionDockerTest.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.druid.common.utils.IdUtils;
+import org.apache.druid.data.input.impl.CsvInputFormat;
+import org.apache.druid.data.input.impl.DimensionsSpec;
+import org.apache.druid.data.input.impl.TimestampSpec;
+import org.apache.druid.indexer.TaskState;
+import org.apache.druid.indexer.TaskStatusPlus;
+import org.apache.druid.indexing.kafka.KafkaIndexTaskModule;
+import org.apache.druid.indexing.kafka.simulate.KafkaResource;
+import org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorIOConfig;
+import org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorSpec;
+import org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorTuningConfig;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.supervisor.SupervisorStatus;
+import org.apache.druid.java.util.common.DateTimes;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.parsers.CloseableIterator;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.indexing.DataSchema;
+import org.apache.druid.testing.embedded.EmbeddedBroker;
+import org.apache.druid.testing.embedded.EmbeddedClusterApis;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.EmbeddedOverlord;
+import org.apache.druid.testing.embedded.EmbeddedRouter;
+import org.apache.druid.testing.embedded.derby.EmbeddedDerbyMetadataResource;
+import org.apache.druid.testing.embedded.indexing.Resources;
+import org.apache.druid.testing.embedded.junit5.EmbeddedClusterTestBase;
+import org.apache.druid.testing.embedded.minio.MinIOStorageResource;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.joda.time.DateTime;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ThreadLocalRandom;
+
+/**
+ * Runs some basic ingestion tests using {@link DruidContainers}.
+ */
+public class IngestionDockerTest extends EmbeddedClusterTestBase
+{
+  static {
+    System.setProperty("druid.testing.docker.image", "apache/druid:35.0.0-SNAPSHOT");
+  }
+
+  // Druid Docker containers
+  protected final DruidContainerResource overlordLeader = DruidContainers.newOverlord().withTestImage();
+  protected final DruidContainerResource coordinator = DruidContainers.newCoordinator().withTestImage();
+  protected final DruidContainerResource historical = DruidContainers.newHistorical().withTestImage();
+  protected final DruidContainerResource broker1 = DruidContainers.newBroker().withTestImage();
+  protected final DruidContainerResource middleManager = DruidContainers
+      .newMiddleManager()
+      .withTestImage()
+      .addProperty("druid.segment.handoff.pollDuration", "PT0.1s");
+
+  // Follower EmbeddedOverlord to watch segment publish events
+  private final EmbeddedOverlord overlordFollower = new EmbeddedOverlord()
+      .addProperty("druid.plaintextPort", "7090")
+      .addProperty("druid.manager.segments.useIncrementalCache", "always")
+      .addProperty("druid.manager.segments.pollDuration", "PT0.1s");
+
+  // Additional EmbeddedBroker to wait for segments to become queryable
+  private final EmbeddedBroker broker2 = new EmbeddedBroker()
+      .addProperty("druid.plaintextPort", "7082");
+
+  private final KafkaResource kafkaServer = new KafkaResource();
+
+  @Override
+  public EmbeddedDruidCluster createCluster()
+  {
+    final EmbeddedDruidCluster cluster = EmbeddedDruidCluster.withZookeeper();
+
+    return cluster
+        .useDruidContainers()
+        .useLatchableEmitter()
+        // Needed for overlordFollower to recognize the KafkaSupervisor type
+        .addExtension(KafkaIndexTaskModule.class)
+        .addResource(new EmbeddedDerbyMetadataResource())
+        .addResource(new MinIOStorageResource())
+        .addResource(kafkaServer)
+        .addCommonProperty(
+            "druid.extensions.loadList",
+            "[\"druid-s3-extensions\", \"druid-kafka-indexing-service\", \"druid-multi-stage-query\"]"
+        )
+        .addResource(coordinator)
+        .addResource(overlordLeader)
+        .addResource(middleManager)
+        .addResource(historical)
+        .addResource(broker1)
+        .addServer(overlordFollower)

Review Comment:
   added.



##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/IngestionDockerTest.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.druid.common.utils.IdUtils;
+import org.apache.druid.data.input.impl.CsvInputFormat;
+import org.apache.druid.data.input.impl.DimensionsSpec;
+import org.apache.druid.data.input.impl.TimestampSpec;
+import org.apache.druid.indexer.TaskState;
+import org.apache.druid.indexer.TaskStatusPlus;
+import org.apache.druid.indexing.kafka.KafkaIndexTaskModule;
+import org.apache.druid.indexing.kafka.simulate.KafkaResource;
+import org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorIOConfig;
+import org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorSpec;
+import org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorTuningConfig;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.supervisor.SupervisorStatus;
+import org.apache.druid.java.util.common.DateTimes;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.parsers.CloseableIterator;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.indexing.DataSchema;
+import org.apache.druid.testing.embedded.EmbeddedBroker;
+import org.apache.druid.testing.embedded.EmbeddedClusterApis;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.EmbeddedOverlord;
+import org.apache.druid.testing.embedded.EmbeddedRouter;
+import org.apache.druid.testing.embedded.derby.EmbeddedDerbyMetadataResource;
+import org.apache.druid.testing.embedded.indexing.Resources;
+import org.apache.druid.testing.embedded.junit5.EmbeddedClusterTestBase;
+import org.apache.druid.testing.embedded.minio.MinIOStorageResource;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.joda.time.DateTime;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ThreadLocalRandom;
+
+/**
+ * Runs some basic ingestion tests using {@link DruidContainers}.
+ */
+public class IngestionDockerTest extends EmbeddedClusterTestBase
+{
+  static {
+    System.setProperty("druid.testing.docker.image", "apache/druid:35.0.0-SNAPSHOT");
+  }
+
+  // Druid Docker containers
+  protected final DruidContainerResource overlordLeader = DruidContainers.newOverlord().withTestImage();

Review Comment:
   added a validation.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.

kfaraz commented on code in PR #18302:
URL: https://github.com/apache/druid/pull/18302#discussion_r2239051329


##########
extensions-core/druid-testcontainers/pom.xml:
##########
@@ -0,0 +1,70 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+  ~ Licensed to the Apache Software Foundation (ASF) under one
+  ~ or more contributor license agreements.  See the NOTICE file
+  ~ distributed with this work for additional information
+  ~ regarding copyright ownership.  The ASF licenses this file
+  ~ to you under the Apache License, Version 2.0 (the
+  ~ "License"); you may not use this file except in compliance
+  ~ with the License.  You may obtain a copy of the License at
+  ~
+  ~   http://www.apache.org/licenses/LICENSE-2.0
+  ~
+  ~ Unless required by applicable law or agreed to in writing,
+  ~ software distributed under the License is distributed on an
+  ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  ~ KIND, either express or implied.  See the License for the
+  ~ specific language governing permissions and limitations
+  ~ under the License.
+  -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
+  <modelVersion>4.0.0</modelVersion>
+
+  <groupId>org.apache.druid.extensions</groupId>
+  <artifactId>druid-testcontainers</artifactId>
+  <name>druid-testcontainers</name>
+  <description>Testcontainers for running Apache Druid services in tests</description>
+
+  <parent>
+    <groupId>org.apache.druid</groupId>
+    <artifactId>druid</artifactId>
+    <version>35.0.0-SNAPSHOT</version>
+    <relativePath>../../pom.xml</relativePath>
+  </parent>
+
+  <!-- This module should have minimal dependencies so that it can be used independently
+   by libraries and codebases outside Druid to run Testcontainers -->
+  <dependencies>
+    <dependency>
+      <groupId>org.apache.druid</groupId>
+      <artifactId>druid-processing</artifactId>

Review Comment:
   Thanks for calling this out, I had similar apprehensions.
   Removed logger and `StringUtils`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.

kfaraz commented on PR #18302:
URL: https://github.com/apache/druid/pull/18302#issuecomment-3140124400

   @gianm , thanks for the feedback.
   
   I have made the following changes:
   - Added a script `run_docker-tests`.
   - This script is used by GHA and can also be used to run all docker tests
   - The script expects an environment variable to be set. Otherwise the tests are not run.
   - While voting on release candidates, committers can just invoke this script with the correct image name. If not available, the image would be downloaded by Testcontainers itself.
   - Docker tests are now disabled by default and will be triggered only in `mvn verify` given that the required system property `-Ddruid.testing.docker.image` is set.
   
   @kgyrtkirk , I think you had also suggested using `verify` phase and adding a dedicated script.
   
   Please let me know if this makes sense.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.

kfaraz commented on code in PR #18302:
URL: https://github.com/apache/druid/pull/18302#discussion_r2244409865


##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/IngestionDockerTest.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.druid.common.utils.IdUtils;
+import org.apache.druid.data.input.impl.CsvInputFormat;
+import org.apache.druid.data.input.impl.DimensionsSpec;
+import org.apache.druid.data.input.impl.TimestampSpec;
+import org.apache.druid.indexer.TaskState;
+import org.apache.druid.indexer.TaskStatusPlus;
+import org.apache.druid.indexing.kafka.KafkaIndexTaskModule;
+import org.apache.druid.indexing.kafka.simulate.KafkaResource;
+import org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorIOConfig;
+import org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorSpec;
+import org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorTuningConfig;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.supervisor.SupervisorStatus;
+import org.apache.druid.java.util.common.DateTimes;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.parsers.CloseableIterator;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.indexing.DataSchema;
+import org.apache.druid.testing.embedded.EmbeddedBroker;
+import org.apache.druid.testing.embedded.EmbeddedClusterApis;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.EmbeddedOverlord;
+import org.apache.druid.testing.embedded.EmbeddedRouter;
+import org.apache.druid.testing.embedded.derby.EmbeddedDerbyMetadataResource;
+import org.apache.druid.testing.embedded.indexing.Resources;
+import org.apache.druid.testing.embedded.junit5.EmbeddedClusterTestBase;
+import org.apache.druid.testing.embedded.minio.MinIOStorageResource;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.joda.time.DateTime;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ThreadLocalRandom;
+
+/**
+ * Runs some basic ingestion tests using {@link DruidContainers}.
+ */
+public class IngestionDockerTest extends EmbeddedClusterTestBase
+{
+  static {
+    System.setProperty("druid.testing.docker.image", "apache/druid:35.0.0-SNAPSHOT");
+  }
+
+  // Druid Docker containers
+  protected final DruidContainerResource overlordLeader = DruidContainers.newOverlord().withTestImage();
+  protected final DruidContainerResource coordinator = DruidContainers.newCoordinator().withTestImage();
+  protected final DruidContainerResource historical = DruidContainers.newHistorical().withTestImage();
+  protected final DruidContainerResource broker1 = DruidContainers.newBroker().withTestImage();
+  protected final DruidContainerResource middleManager = DruidContainers
+      .newMiddleManager()
+      .withTestImage()
+      .addProperty("druid.segment.handoff.pollDuration", "PT0.1s");
+
+  // Follower EmbeddedOverlord to watch segment publish events
+  private final EmbeddedOverlord overlordFollower = new EmbeddedOverlord()
+      .addProperty("druid.plaintextPort", "7090")
+      .addProperty("druid.manager.segments.useIncrementalCache", "always")
+      .addProperty("druid.manager.segments.pollDuration", "PT0.1s");
+
+  // Additional EmbeddedBroker to wait for segments to become queryable
+  private final EmbeddedBroker broker2 = new EmbeddedBroker()
+      .addProperty("druid.plaintextPort", "7082");
+
+  private final KafkaResource kafkaServer = new KafkaResource();
+
+  @Override
+  public EmbeddedDruidCluster createCluster()
+  {
+    final EmbeddedDruidCluster cluster = EmbeddedDruidCluster.withZookeeper();
+
+    return cluster
+        .useDruidContainers()
+        .useLatchableEmitter()
+        // Needed for overlordFollower to recognize the KafkaSupervisor type
+        .addExtension(KafkaIndexTaskModule.class)
+        .addResource(new EmbeddedDerbyMetadataResource())
+        .addResource(new MinIOStorageResource())
+        .addResource(kafkaServer)
+        .addCommonProperty(
+            "druid.extensions.loadList",
+            "[\"druid-s3-extensions\", \"druid-kafka-indexing-service\", \"druid-multi-stage-query\"]"
+        )
+        .addResource(coordinator)
+        .addResource(overlordLeader)
+        .addResource(middleManager)
+        .addResource(historical)
+        .addResource(broker1)
+        .addServer(overlordFollower)
+        .addServer(broker2)
+        .addServer(new EmbeddedRouter());
+  }
+
+  @Test
+  public void test_runIndexTask_andKillData()
+  {
+

Review Comment:
   added



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "github-advanced-security[bot] (via GitHub)" <gi...@apache.org>.

github-advanced-security[bot] commented on code in PR #18302:
URL: https://github.com/apache/druid/pull/18302#discussion_r2243421277


##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/IngestionDockerTest.java:
##########
@@ -0,0 +1,369 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.druid.common.utils.IdUtils;
+import org.apache.druid.data.input.impl.CsvInputFormat;
+import org.apache.druid.data.input.impl.DimensionsSpec;
+import org.apache.druid.data.input.impl.TimestampSpec;
+import org.apache.druid.indexer.TaskState;
+import org.apache.druid.indexer.TaskStatusPlus;
+import org.apache.druid.indexing.common.task.IndexTask;
+import org.apache.druid.indexing.common.task.TaskBuilder;
+import org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexSupervisorTask;
+import org.apache.druid.indexing.kafka.KafkaIndexTaskModule;
+import org.apache.druid.indexing.kafka.simulate.KafkaResource;
+import org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorIOConfig;
+import org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorSpec;
+import org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorTuningConfig;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.supervisor.SupervisorStatus;
+import org.apache.druid.java.util.common.DateTimes;
+import org.apache.druid.java.util.common.Intervals;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.parsers.CloseableIterator;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.query.http.SqlTaskStatus;
+import org.apache.druid.segment.indexing.DataSchema;
+import org.apache.druid.testing.DruidCommand;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.EmbeddedOverlord;
+import org.apache.druid.testing.embedded.derby.EmbeddedDerbyMetadataResource;
+import org.apache.druid.testing.embedded.emitter.LatchableEmitterModule;
+import org.apache.druid.testing.embedded.indexing.MoreResources;
+import org.apache.druid.testing.embedded.indexing.Resources;
+import org.apache.druid.testing.embedded.junit5.EmbeddedClusterTestBase;
+import org.apache.druid.testing.embedded.minio.MinIOStorageResource;
+import org.apache.druid.testing.embedded.msq.EmbeddedMSQApis;
+import org.apache.druid.testing.embedded.server.EmbeddedEventCollector;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.joda.time.DateTime;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ThreadLocalRandom;
+
+/**
+ * Runs some basic ingestion tests using {@code DruidContainers}.
+ * <p>
+ * This test verifies functionality of Druid Docker images using
+ * {@link DruidContainerResource}. However, the underlying cluster also uses
+ * embedded servers either to provide visibility into the cluster state or to
+ * test backwards compatibility of a given feature.
+ */
+public class IngestionDockerTest extends EmbeddedClusterTestBase
+{
+  // Druid Docker containers
+  protected final DruidContainerResource overlordLeader =
+      new DruidContainerResource(DruidCommand.OVERLORD).usingTestImage();
+  protected final DruidContainerResource coordinator =
+      new DruidContainerResource(DruidCommand.COORDINATOR).usingTestImage();
+  protected final DruidContainerResource historical =
+      new DruidContainerResource(DruidCommand.HISTORICAL).usingTestImage();
+  protected final DruidContainerResource broker =
+      new DruidContainerResource(DruidCommand.BROKER)
+          .addProperty("druid.sql.planner.metadataRefreshPeriod", "PT1s")
+          .usingTestImage();
+  protected final DruidContainerResource router =
+      new DruidContainerResource(DruidCommand.ROUTER).usingTestImage();
+  protected final DruidContainerResource middleManager =
+      new DruidContainerResource(DruidCommand.MIDDLE_MANAGER)
+          .addProperty("druid.segment.handoff.pollDuration", "PT0.1s")
+          .usingTestImage();
+
+  // Follower EmbeddedOverlord to help serialize Task and Supervisor payloads
+  private final EmbeddedOverlord overlordFollower =
+      new EmbeddedOverlord().addProperty("druid.plaintextPort", "7090");
+
+  // Event collector to watch for metric events
+  private final EmbeddedEventCollector eventCollector = new EmbeddedEventCollector()
+      .addProperty("druid.emitter", "latching")
+      .addProperty("druid.server.http.numThreads", "20");
+
+  private final KafkaResource kafkaServer = new KafkaResource();
+
+  @Override
+  public EmbeddedDruidCluster createCluster()
+  {
+    return EmbeddedDruidCluster
+        .withZookeeper()
+        .useDruidContainers()
+        // Needed for overlordFollower to recognize the KafkaSupervisor type
+        .addExtensions(KafkaIndexTaskModule.class, LatchableEmitterModule.class)
+        .addResource(new EmbeddedDerbyMetadataResource())
+        .addResource(new MinIOStorageResource())
+        .addResource(kafkaServer)
+        .addCommonProperty(
+            "druid.extensions.loadList",
+            "[\"druid-s3-extensions\", \"druid-kafka-indexing-service\", \"druid-multi-stage-query\"]"
+        )
+        .addCommonProperty("druid.emitter", "http")
+        .addCommonProperty("druid.emitter.http.recipientBaseUrl", eventCollector.getMetricsUrl())
+        .addCommonProperty("druid.emitter.http.flushMillis", "500")
+        .addResource(coordinator)
+        .addResource(overlordLeader)
+        .addResource(middleManager)
+        .addResource(historical)
+        .addResource(broker)
+        .addResource(router)
+        .addServer(overlordFollower)
+        .addServer(eventCollector);
+  }
+
+  @BeforeEach
+  @AfterEach
+  public void verifyOverlordLeader()
+  {
+    Assertions.assertFalse(
+        overlordFollower.bindings().overlordLeaderSelector().isLeader()
+    );
+  }
+
+  @Test
+  public void test_runIndexTask_andKillData()
+  {
+    final int numSegments = 10;
+
+    // Run an 'index' task and verify the ingested data
+    final String taskId = IdUtils.getRandomId();
+    final IndexTask task = MoreResources.Task.BASIC_INDEX.get().dataSource(dataSource).withId(taskId);
+    cluster.callApi().onLeaderOverlord(o -> o.runTask(taskId, task));
+    cluster.callApi().waitForTaskToSucceed(taskId, eventCollector.latchableEmitter());
+
+    verifyUsedSegmentCount(numSegments);
+    waitForSegmentsToBeQueryable(numSegments);
+
+    cluster.callApi().verifySqlQuery("SELECT COUNT(*) FROM sys.segments WHERE datasource='%s'", dataSource, "10");
+    cluster.callApi().verifySqlQuery("SELECT * FROM %s", dataSource, Resources.InlineData.CSV_10_DAYS);
+
+    // Mark all segments as unused and verify state
+    Assertions.assertEquals(
+        numSegments,
+        markSegmentsAsUnused(dataSource)
+    );
+    verifyUsedSegmentCount(0);
+    eventCollector.latchableEmitter().waitForEventAggregate(
+        event -> event.hasMetricName("segment/loadQueue/success")
+                      .hasDimension(DruidMetrics.DESCRIPTION, "DROP")
+                      .hasService("druid/coordinator"),
+        agg -> agg.hasSumAtLeast(numSegments)
+    );
+    cluster.callApi().verifySqlQuery("SELECT * FROM sys.segments WHERE datasource='%s'", dataSource, "");
+
+    // Kill all unused segments
+    final String killTaskId = cluster.callApi().onLeaderOverlord(
+        o -> o.runKillTask(IdUtils.getRandomId(), dataSource, Intervals.ETERNITY, null, null, null)
+    );
+    cluster.callApi().waitForTaskToSucceed(killTaskId, eventCollector.latchableEmitter());
+
+    eventCollector.latchableEmitter().waitForEventAggregate(
+        event -> event.hasMetricName("segment/nuked/bytes")
+                      .hasService("druid/overlord"),
+        agg -> agg.hasCountAtLeast(numSegments)
+    );
+  }
+
+  @Test
+  public void test_runIndexParallelTask_andCompactData()
+  {
+    final int numSegments = 10;

Review Comment:
   ## Unread local variable
   
   Variable 'int numSegments' is never read.
   
   [Show more details](https://github.com/apache/druid/security/code-scanning/9939)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.

kfaraz commented on PR #18302:
URL: https://github.com/apache/druid/pull/18302#issuecomment-3140125089

   @gianm , I also needed to include some other changes to provide visibility into the containers.
   
   - Converted a `CliCustomNodeRole` to a `CliEventCollector`
   - This is a custom test-only Druid service that just collects events over HTTP
   - The events are then forwarded to a `LatchableEmitter` which allows watching for certain events
   - Added a `CustomNodeRoleDockerTest` which supersedes the `ITHighAvailabilityTest`
   - Removed code related to the old high availability tests.
   
   I hope these extra changes have not made the PR too bloated.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.

kfaraz commented on code in PR #18302:
URL: https://github.com/apache/druid/pull/18302#discussion_r2236981333


##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/DruidContainerResource.java:
##########
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import org.apache.druid.java.util.common.FileUtils;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.testing.DruidCommand;
+import org.apache.druid.testing.DruidContainer;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.TestcontainerResource;
+import org.testcontainers.containers.BindMode;
+import org.testcontainers.containers.wait.strategy.Wait;
+import org.testcontainers.utility.DockerImageName;
+
+import java.io.File;
+import java.nio.file.Files;
+import java.nio.file.attribute.PosixFilePermissions;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+
+/**
+ * {@link TestcontainerResource} to run Druid services.
+ * Currently, only core extensions can be used out-of-the-box with these containers
+ * such as {@code druid-s3-extensions} or {@code postgresql-metadata-storage},
+ * simply by adding them to {@code druid.extensions.loadList}.
+ * <p>
+ * {@link DruidContainers} should be used only for testing backward compatiblity
+ * or a Docker-specific feature. For all other testing needs, use plain old
+ * {@code EmbeddedDruidServer} as they are much faster, allow easy debugging and
+ * do not require downloading any images.
+ */
+public class DruidContainerResource extends TestcontainerResource<DruidContainer>
+{
+  /**
+   * Java system property to specify the name of the Docker test image.
+   */
+  public static final String PROPERTY_TEST_IMAGE = "druid.testing.docker.image";
+
+  private static final Logger log = new Logger(DruidContainerResource.class);
+
+  /**
+   * Forbidden server properties that may be used by EmbeddedDruidServers but
+   * interfere with the functioning of DruidContainer-based services.
+   */
+  private static final Set<String> FORBIDDEN_PROPERTIES = Set.of(
+      "druid.extensions.modulesForEmbeddedTests",
+      "druid.emitter"
+  );
+
+  /**
+   * A static incremental ID is used instead of a random number to ensure that
+   * tests are more deterministic and easier to debug.
+   */
+  private static final AtomicInteger SERVER_ID = new AtomicInteger(0);
+
+  private final String name;
+  private final DruidCommand command;
+  private final Map<String, String> properties = new HashMap<>();
+  private final List<Integer> taskPorts = new ArrayList<>();
+
+  private int servicePort;
+  private DockerImageName imageName;
+  private EmbeddedDruidCluster cluster;
+
+  private String containerDirectory;
+
+  private MountedDir indexerLogsDeepStorageDirectory;
+  private MountedDir serviceLogsDirectory;
+  private MountedDir segmentDeepStorageDirectory;
+
+  DruidContainerResource(DruidCommand command)
+  {
+    this.name = StringUtils.format(
+        "container_%s_%d",
+        command.getName(),
+        SERVER_ID.incrementAndGet()
+    );
+    this.command = command;
+
+    Integer[] exposedPorts = command.getExposedPorts();
+    servicePort = exposedPorts[0];
+    taskPorts.addAll(Arrays.asList(exposedPorts).subList(1, exposedPorts.length));
+  }
+
+  public DruidContainerResource withPort(int port)
+  {
+    this.servicePort = port;
+    return this;
+  }
+
+  /**
+   * Used to bind the task ports on middle managers.
+   */
+  public DruidContainerResource withTaskPorts(int startPort, int workerCapacity)
+  {
+    taskPorts.clear();
+    for (int i = 0; i < workerCapacity; ++i) {
+      taskPorts.add(startPort + i);
+    }
+    addProperty("druid.worker.capacity", String.valueOf(workerCapacity));
+    addProperty("druid.indexer.runner.startPort", String.valueOf(startPort));
+    addProperty("druid.indexer.runner.endPort", String.valueOf(startPort + workerCapacity));
+
+    return this;
+  }
+
+  public DruidContainerResource withImage(DockerImageName imageName)
+  {
+    this.imageName = imageName;
+    return this;
+  }
+
+  /**
+   * Uses the Docker test image specified by the system property
+   * {@link #PROPERTY_TEST_IMAGE} for this container.
+   */
+  public DruidContainerResource withTestImage()
+  {
+    String imageName = Objects.requireNonNull(
+        System.getProperty(PROPERTY_TEST_IMAGE),
+        StringUtils.format("System property[%s] is not set", PROPERTY_TEST_IMAGE)
+    );
+    return withImage(DockerImageName.parse(imageName));
+  }
+
+  public DruidContainerResource addProperty(String key, String value)
+  {
+    properties.put(key, value);
+    return this;
+  }
+
+  @Override
+  public void beforeStart(EmbeddedDruidCluster cluster)
+  {
+    this.cluster = cluster;
+
+    // Mount directories used by the entire cluster (including embedded servers)
+    this.segmentDeepStorageDirectory = new MountedDir(
+        "/tmp/druid/deep-store",
+        cluster.getTestFolder().getOrCreateFolder("deep-store").getAbsolutePath()
+    );
+    this.indexerLogsDeepStorageDirectory = new MountedDir(
+        "/tmp/druid/indexer-logs",
+        cluster.getTestFolder().getOrCreateFolder("indexer-logs").getAbsolutePath()
+    );
+
+    // Mount directories used by this container for easier debugging with service logs
+    this.containerDirectory = cluster.getTestFolder().getOrCreateFolder(name).getAbsolutePath();
+    this.serviceLogsDirectory = new MountedDir(
+        "/opt/druid/log",
+        containerDirectory + "/log"
+    );
+
+    // Create the log directory upfront to avoid permission issues
+    createLogDirectory(new File(containerDirectory, "log"));
+  }
+
+  @Override
+  protected DruidContainer createContainer()
+  {
+    final List<Integer> exposedPorts = new ArrayList<>(taskPorts);
+    exposedPorts.add(servicePort);
+    log.info(
+        "Starting Druid container[%s] on port[%d] with mounted directory[%s] and exposed ports[%s].",
+        name, servicePort, containerDirectory, exposedPorts
+    );
+
+    final DruidContainer container = new DruidContainer(command, imageName)
+        .withExposedPorts(exposedPorts.toArray(new Integer[0]))
+        .withCommonProperties(getCommonProperties())
+        .withServiceProperties(getServerProperties())
+        .withFileSystemBind(
+            segmentDeepStorageDirectory.hostPath,
+            segmentDeepStorageDirectory.containerPath,
+            BindMode.READ_WRITE
+        )
+        .withFileSystemBind(
+            indexerLogsDeepStorageDirectory.hostPath,
+            indexerLogsDeepStorageDirectory.containerPath,
+            BindMode.READ_WRITE
+        )
+        .withFileSystemBind(
+            serviceLogsDirectory.hostPath,
+            serviceLogsDirectory.containerPath,
+            BindMode.READ_WRITE
+        )
+        .withEnv(
+            Map.of(
+                "DRUID_SET_HOST_IP", "0",
+                "DRUID_SET_HOST", "0"
+            )
+        )
+        .waitingFor(Wait.forHttp("/status/health").forPort(servicePort));
+
+    // Bind the ports statically (rather than using a mapped port) to the same
+    // value used in `druid.plaintextPort` to make this node discoverable
+    // by other services (both embedded and dockerized).
+    List<String> portBindings = exposedPorts.stream().map(
+        port -> StringUtils.format("%d:%d", port, port)
+    ).collect(Collectors.toList());
+    container.setPortBindings(portBindings);
+
+    return container;
+  }
+
+  /**
+   * Writes the common properties to a file.
+   *
+   * @return Name of the file.
+   */
+  private Properties getCommonProperties()
+  {
+    final Properties commonProperties = new Properties();
+    commonProperties.putAll(cluster.getCommonProperties());
+    FORBIDDEN_PROPERTIES.forEach(commonProperties::remove);
+
+    commonProperties.setProperty(
+        "druid.storage.storageDirectory",
+        segmentDeepStorageDirectory.containerPath
+    );
+    commonProperties.setProperty(
+        "druid.indexer.logs.directory",
+        indexerLogsDeepStorageDirectory.containerPath
+    );
+
+    log.info(
+        "Writing common properties for Druid container[%s]: [%s]",
+        name, commonProperties
+    );
+
+    return commonProperties;
+  }
+
+  /**
+   * Writes the server properties to a file.
+   *
+   * @return Name of the file.
+   */
+  private Properties getServerProperties()
+  {
+    FORBIDDEN_PROPERTIES.forEach(properties::remove);
+    addProperty("druid.host", EmbeddedDruidCluster.getDefaultHost());
+    addProperty("druid.plaintextPort", String.valueOf(servicePort));
+
+    final Properties serverProperties = new Properties();
+    serverProperties.putAll(properties);
+
+    log.info(
+        "Writing runtime properties for Druid container[%s]: [%s]",
+        name, serverProperties
+    );
+
+    return serverProperties;
+  }
+
+  private static void createLogDirectory(File dir)
+  {
+    try {
+      FileUtils.mkdirp(dir);
+      Files.setPosixFilePermissions(dir.toPath(), PosixFilePermissions.fromString("rwxrwxrwx"));
+    }
+    catch (Exception e) {
+      throw new RuntimeException(e);
+    }
+  }
+
+  @Override
+  public String toString()
+  {
+    return name;
+  }
+
+  private static class MountedDir
+  {
+    final String hostPath;
+    final String containerPath;

Review Comment:
   used `File` instead.



##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/DruidContainerResource.java:
##########
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import org.apache.druid.java.util.common.FileUtils;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.testing.DruidCommand;
+import org.apache.druid.testing.DruidContainer;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.TestcontainerResource;
+import org.testcontainers.containers.BindMode;
+import org.testcontainers.containers.wait.strategy.Wait;
+import org.testcontainers.utility.DockerImageName;
+
+import java.io.File;
+import java.nio.file.Files;
+import java.nio.file.attribute.PosixFilePermissions;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+
+/**
+ * {@link TestcontainerResource} to run Druid services.
+ * Currently, only core extensions can be used out-of-the-box with these containers
+ * such as {@code druid-s3-extensions} or {@code postgresql-metadata-storage},
+ * simply by adding them to {@code druid.extensions.loadList}.
+ * <p>
+ * {@link DruidContainers} should be used only for testing backward compatiblity
+ * or a Docker-specific feature. For all other testing needs, use plain old
+ * {@code EmbeddedDruidServer} as they are much faster, allow easy debugging and
+ * do not require downloading any images.
+ */
+public class DruidContainerResource extends TestcontainerResource<DruidContainer>
+{
+  /**
+   * Java system property to specify the name of the Docker test image.
+   */
+  public static final String PROPERTY_TEST_IMAGE = "druid.testing.docker.image";
+
+  private static final Logger log = new Logger(DruidContainerResource.class);
+
+  /**
+   * Forbidden server properties that may be used by EmbeddedDruidServers but
+   * interfere with the functioning of DruidContainer-based services.
+   */
+  private static final Set<String> FORBIDDEN_PROPERTIES = Set.of(
+      "druid.extensions.modulesForEmbeddedTests",
+      "druid.emitter"
+  );
+
+  /**
+   * A static incremental ID is used instead of a random number to ensure that
+   * tests are more deterministic and easier to debug.
+   */
+  private static final AtomicInteger SERVER_ID = new AtomicInteger(0);
+
+  private final String name;
+  private final DruidCommand command;
+  private final Map<String, String> properties = new HashMap<>();
+  private final List<Integer> taskPorts = new ArrayList<>();
+
+  private int servicePort;
+  private DockerImageName imageName;
+  private EmbeddedDruidCluster cluster;
+
+  private String containerDirectory;

Review Comment:
   used `File` instead.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.

kfaraz commented on code in PR #18302:
URL: https://github.com/apache/druid/pull/18302#discussion_r2236098861


##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/DruidContainers.java:
##########
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import org.apache.druid.testing.DruidCommand;
+
+/**
+ * Factory for {@link DruidContainerResource} that can run specific services.
+ *
+ * @see #newOverlord()
+ * @see #newCoordinator()
+ */
+public final class DruidContainers
+{
+  private DruidContainers()
+  {
+    // no instantiation
+  }
+
+  /**
+   * Creates a new {@link DruidContainerResource} to run a Coordinator node.
+   */
+  public static DruidContainerResource newCoordinator()
+  {
+    return new DruidContainerResource(DruidCommand.COORDINATOR)
+        .addProperty("druid.coordinator.startDelay", "PT0.1S")
+        .addProperty("druid.coordinator.period", "PT0.5S")
+        .addProperty("druid.manager.segments.pollDuration", "PT0.1S");
+  }
+
+  /**
+   * Creates a new {@link DruidContainerResource} to run an Overlord node.
+   */
+  public static DruidContainerResource newOverlord()
+  {
+    // Keep a small sync timeout so that Peons and Indexers are not stuck
+    // handling a change request when Overlord has already shutdown
+    return new DruidContainerResource(DruidCommand.OVERLORD)
+        .addProperty("druid.indexer.storage.type", "metadata")
+        .addProperty("druid.indexer.queue.startDelay", "PT0S")
+        .addProperty("druid.indexer.queue.restartDelay", "PT0S")
+        .addProperty("druid.indexer.runner.syncRequestTimeout", "PT1S");
+  }
+
+  /**
+   * Creates a new {@link DruidContainerResource} to run an Indexer node.
+   */
+  public static DruidContainerResource newIndexer()
+  {
+    return new DruidContainerResource(DruidCommand.INDEXER)
+        .addProperty("druid.lookup.enableLookupSyncOnStartup", "false")
+        .addProperty("druid.processing.buffer.sizeBytes", "50MiB")
+        .addProperty("druid.processing.numMergeBuffers", "2")
+        .addProperty("druid.processing.numThreads", "5");
+  }

Review Comment:
   Yes, that's true. `DruidCommand` will evolve over time.
   
   There are 2 parts to this PR:
   
   1. Add module `druid-testcontainers`.
     - This module contains `DruidContainer` which extends `GenericContainer` from `Testcontainers` library.
     - The aim with this module is that it can be used _independently_ as a dependency by anyone in the wild trying to run Druid as a Testcontainer.
     - It is not really mature enough to do it right now. But I am hoping that by Druid 35, it will be well-cooked enough for independent use.
     - `DruidCommand` will be used by anyone running a `DruidContainer`. So it contains any property that are needed out of the box to run any Druid service.
   
   2. Add `DruidContainerResource` which uses the `DruidContainer` in our own embedded-tests framework.
     - This is for use within Apache Druid repo only.
     - Any property needed to run a DruidContainer within embedded tests has been kept in the `DruidContainers` factory class.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.

kfaraz commented on code in PR #18302:
URL: https://github.com/apache/druid/pull/18302#discussion_r2236098861


##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/DruidContainers.java:
##########
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import org.apache.druid.testing.DruidCommand;
+
+/**
+ * Factory for {@link DruidContainerResource} that can run specific services.
+ *
+ * @see #newOverlord()
+ * @see #newCoordinator()
+ */
+public final class DruidContainers
+{
+  private DruidContainers()
+  {
+    // no instantiation
+  }
+
+  /**
+   * Creates a new {@link DruidContainerResource} to run a Coordinator node.
+   */
+  public static DruidContainerResource newCoordinator()
+  {
+    return new DruidContainerResource(DruidCommand.COORDINATOR)
+        .addProperty("druid.coordinator.startDelay", "PT0.1S")
+        .addProperty("druid.coordinator.period", "PT0.5S")
+        .addProperty("druid.manager.segments.pollDuration", "PT0.1S");
+  }
+
+  /**
+   * Creates a new {@link DruidContainerResource} to run an Overlord node.
+   */
+  public static DruidContainerResource newOverlord()
+  {
+    // Keep a small sync timeout so that Peons and Indexers are not stuck
+    // handling a change request when Overlord has already shutdown
+    return new DruidContainerResource(DruidCommand.OVERLORD)
+        .addProperty("druid.indexer.storage.type", "metadata")
+        .addProperty("druid.indexer.queue.startDelay", "PT0S")
+        .addProperty("druid.indexer.queue.restartDelay", "PT0S")
+        .addProperty("druid.indexer.runner.syncRequestTimeout", "PT1S");
+  }
+
+  /**
+   * Creates a new {@link DruidContainerResource} to run an Indexer node.
+   */
+  public static DruidContainerResource newIndexer()
+  {
+    return new DruidContainerResource(DruidCommand.INDEXER)
+        .addProperty("druid.lookup.enableLookupSyncOnStartup", "false")
+        .addProperty("druid.processing.buffer.sizeBytes", "50MiB")
+        .addProperty("druid.processing.numMergeBuffers", "2")
+        .addProperty("druid.processing.numThreads", "5");
+  }

Review Comment:
   Yes, that's true. `DruidCommand` will evolve over time.
   
   There are 2 parts to this PR:
   
   1. Add module `druid-testcontainers`.
     - This module contains `DruidContainer` which extends `GenericContainer` from `Testcontainers` library.
     - The aim with this module is that it can be used _independently_ as a dependency by anyone in the wild trying to run Druid as a Testcontainer.
     - It is not really mature enough to do it right now. But I am hoping that by Druid 35, it will be well-cooked enough for independent use.
     - `DruidCommand` will be used by anyone running a `DruidContainer`. So it contains any property that are needed out of the box to run any Druid service.
   
   2. Add `DruidContainerResource` which uses the `DruidContainer` in our own embedded-tests framework.
     - This is for use within Apache Druid repo only.
     - Any property needed to run a DruidContainer within embedded tests has been kept in the `DruidContainers` factory class.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.

kfaraz commented on code in PR #18302:
URL: https://github.com/apache/druid/pull/18302#discussion_r2237416340


##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/DruidContainerResource.java:
##########
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import org.apache.druid.java.util.common.FileUtils;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.testing.DruidCommand;
+import org.apache.druid.testing.DruidContainer;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.TestcontainerResource;
+import org.testcontainers.containers.BindMode;
+import org.testcontainers.containers.wait.strategy.Wait;
+import org.testcontainers.utility.DockerImageName;
+
+import java.io.File;
+import java.nio.file.Files;
+import java.nio.file.attribute.PosixFilePermissions;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+
+/**
+ * {@link TestcontainerResource} to run Druid services.
+ * Currently, only core extensions can be used out-of-the-box with these containers
+ * such as {@code druid-s3-extensions} or {@code postgresql-metadata-storage},
+ * simply by adding them to {@code druid.extensions.loadList}.
+ * <p>
+ * {@link DruidContainers} should be used only for testing backward compatiblity
+ * or a Docker-specific feature. For all other testing needs, use plain old
+ * {@code EmbeddedDruidServer} as they are much faster, allow easy debugging and
+ * do not require downloading any images.
+ */
+public class DruidContainerResource extends TestcontainerResource<DruidContainer>
+{
+  /**
+   * Java system property to specify the name of the Docker test image.
+   */
+  public static final String PROPERTY_TEST_IMAGE = "druid.testing.docker.image";
+
+  private static final Logger log = new Logger(DruidContainerResource.class);
+
+  /**
+   * Forbidden server properties that may be used by EmbeddedDruidServers but
+   * interfere with the functioning of DruidContainer-based services.
+   */
+  private static final Set<String> FORBIDDEN_PROPERTIES = Set.of(
+      "druid.extensions.modulesForEmbeddedTests",
+      "druid.emitter"
+  );
+
+  /**
+   * A static incremental ID is used instead of a random number to ensure that
+   * tests are more deterministic and easier to debug.
+   */
+  private static final AtomicInteger SERVER_ID = new AtomicInteger(0);
+
+  private final String name;
+  private final DruidCommand command;
+  private final Map<String, String> properties = new HashMap<>();
+  private final List<Integer> taskPorts = new ArrayList<>();
+
+  private int servicePort;
+  private DockerImageName imageName;
+  private EmbeddedDruidCluster cluster;
+
+  private String containerDirectory;
+
+  private MountedDir indexerLogsDeepStorageDirectory;
+  private MountedDir serviceLogsDirectory;
+  private MountedDir segmentDeepStorageDirectory;
+
+  DruidContainerResource(DruidCommand command)
+  {
+    this.name = StringUtils.format(
+        "container_%s_%d",
+        command.getName(),
+        SERVER_ID.incrementAndGet()
+    );
+    this.command = command;
+
+    Integer[] exposedPorts = command.getExposedPorts();
+    servicePort = exposedPorts[0];
+    taskPorts.addAll(Arrays.asList(exposedPorts).subList(1, exposedPorts.length));
+  }
+
+  public DruidContainerResource withPort(int port)
+  {
+    this.servicePort = port;
+    return this;
+  }
+
+  /**
+   * Used to bind the task ports on middle managers.
+   */
+  public DruidContainerResource withTaskPorts(int startPort, int workerCapacity)
+  {
+    taskPorts.clear();
+    for (int i = 0; i < workerCapacity; ++i) {
+      taskPorts.add(startPort + i);
+    }
+    addProperty("druid.worker.capacity", String.valueOf(workerCapacity));
+    addProperty("druid.indexer.runner.startPort", String.valueOf(startPort));
+    addProperty("druid.indexer.runner.endPort", String.valueOf(startPort + workerCapacity));
+
+    return this;
+  }
+
+  public DruidContainerResource withImage(DockerImageName imageName)
+  {
+    this.imageName = imageName;
+    return this;
+  }
+
+  /**
+   * Uses the Docker test image specified by the system property
+   * {@link #PROPERTY_TEST_IMAGE} for this container.
+   */
+  public DruidContainerResource withTestImage()
+  {
+    String imageName = Objects.requireNonNull(
+        System.getProperty(PROPERTY_TEST_IMAGE),
+        StringUtils.format("System property[%s] is not set", PROPERTY_TEST_IMAGE)
+    );
+    return withImage(DockerImageName.parse(imageName));
+  }
+
+  public DruidContainerResource addProperty(String key, String value)
+  {
+    properties.put(key, value);
+    return this;
+  }
+
+  @Override
+  public void beforeStart(EmbeddedDruidCluster cluster)
+  {
+    this.cluster = cluster;
+
+    // Mount directories used by the entire cluster (including embedded servers)
+    this.segmentDeepStorageDirectory = new MountedDir(
+        "/tmp/druid/deep-store",
+        cluster.getTestFolder().getOrCreateFolder("deep-store").getAbsolutePath()
+    );
+    this.indexerLogsDeepStorageDirectory = new MountedDir(
+        "/tmp/druid/indexer-logs",
+        cluster.getTestFolder().getOrCreateFolder("indexer-logs").getAbsolutePath()
+    );
+
+    // Mount directories used by this container for easier debugging with service logs
+    this.containerDirectory = cluster.getTestFolder().getOrCreateFolder(name).getAbsolutePath();
+    this.serviceLogsDirectory = new MountedDir(
+        "/opt/druid/log",
+        containerDirectory + "/log"
+    );
+
+    // Create the log directory upfront to avoid permission issues
+    createLogDirectory(new File(containerDirectory, "log"));
+  }
+
+  @Override
+  protected DruidContainer createContainer()
+  {
+    final List<Integer> exposedPorts = new ArrayList<>(taskPorts);
+    exposedPorts.add(servicePort);
+    log.info(
+        "Starting Druid container[%s] on port[%d] with mounted directory[%s] and exposed ports[%s].",
+        name, servicePort, containerDirectory, exposedPorts
+    );
+
+    final DruidContainer container = new DruidContainer(command, imageName)
+        .withExposedPorts(exposedPorts.toArray(new Integer[0]))

Review Comment:
   updated.



##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/DruidContainerResource.java:
##########
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import org.apache.druid.java.util.common.FileUtils;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.testing.DruidCommand;
+import org.apache.druid.testing.DruidContainer;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.TestcontainerResource;
+import org.testcontainers.containers.BindMode;
+import org.testcontainers.containers.wait.strategy.Wait;
+import org.testcontainers.utility.DockerImageName;
+
+import java.io.File;
+import java.nio.file.Files;
+import java.nio.file.attribute.PosixFilePermissions;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+
+/**
+ * {@link TestcontainerResource} to run Druid services.
+ * Currently, only core extensions can be used out-of-the-box with these containers
+ * such as {@code druid-s3-extensions} or {@code postgresql-metadata-storage},
+ * simply by adding them to {@code druid.extensions.loadList}.
+ * <p>
+ * {@link DruidContainers} should be used only for testing backward compatiblity
+ * or a Docker-specific feature. For all other testing needs, use plain old
+ * {@code EmbeddedDruidServer} as they are much faster, allow easy debugging and
+ * do not require downloading any images.
+ */
+public class DruidContainerResource extends TestcontainerResource<DruidContainer>
+{
+  /**
+   * Java system property to specify the name of the Docker test image.
+   */
+  public static final String PROPERTY_TEST_IMAGE = "druid.testing.docker.image";
+
+  private static final Logger log = new Logger(DruidContainerResource.class);
+
+  /**
+   * Forbidden server properties that may be used by EmbeddedDruidServers but
+   * interfere with the functioning of DruidContainer-based services.
+   */
+  private static final Set<String> FORBIDDEN_PROPERTIES = Set.of(
+      "druid.extensions.modulesForEmbeddedTests",
+      "druid.emitter"
+  );
+
+  /**
+   * A static incremental ID is used instead of a random number to ensure that
+   * tests are more deterministic and easier to debug.
+   */
+  private static final AtomicInteger SERVER_ID = new AtomicInteger(0);
+
+  private final String name;
+  private final DruidCommand command;
+  private final Map<String, String> properties = new HashMap<>();
+  private final List<Integer> taskPorts = new ArrayList<>();
+
+  private int servicePort;
+  private DockerImageName imageName;
+  private EmbeddedDruidCluster cluster;
+
+  private String containerDirectory;
+
+  private MountedDir indexerLogsDeepStorageDirectory;
+  private MountedDir serviceLogsDirectory;
+  private MountedDir segmentDeepStorageDirectory;
+
+  DruidContainerResource(DruidCommand command)
+  {
+    this.name = StringUtils.format(
+        "container_%s_%d",
+        command.getName(),
+        SERVER_ID.incrementAndGet()
+    );
+    this.command = command;
+
+    Integer[] exposedPorts = command.getExposedPorts();
+    servicePort = exposedPorts[0];
+    taskPorts.addAll(Arrays.asList(exposedPorts).subList(1, exposedPorts.length));
+  }
+
+  public DruidContainerResource withPort(int port)
+  {
+    this.servicePort = port;
+    return this;
+  }
+
+  /**
+   * Used to bind the task ports on middle managers.
+   */
+  public DruidContainerResource withTaskPorts(int startPort, int workerCapacity)
+  {
+    taskPorts.clear();
+    for (int i = 0; i < workerCapacity; ++i) {
+      taskPorts.add(startPort + i);
+    }
+    addProperty("druid.worker.capacity", String.valueOf(workerCapacity));
+    addProperty("druid.indexer.runner.startPort", String.valueOf(startPort));
+    addProperty("druid.indexer.runner.endPort", String.valueOf(startPort + workerCapacity));
+
+    return this;
+  }
+
+  public DruidContainerResource withImage(DockerImageName imageName)
+  {
+    this.imageName = imageName;
+    return this;
+  }
+
+  /**
+   * Uses the Docker test image specified by the system property
+   * {@link #PROPERTY_TEST_IMAGE} for this container.
+   */
+  public DruidContainerResource withTestImage()
+  {
+    String imageName = Objects.requireNonNull(
+        System.getProperty(PROPERTY_TEST_IMAGE),
+        StringUtils.format("System property[%s] is not set", PROPERTY_TEST_IMAGE)
+    );
+    return withImage(DockerImageName.parse(imageName));
+  }
+
+  public DruidContainerResource addProperty(String key, String value)
+  {
+    properties.put(key, value);
+    return this;
+  }
+
+  @Override
+  public void beforeStart(EmbeddedDruidCluster cluster)
+  {
+    this.cluster = cluster;
+
+    // Mount directories used by the entire cluster (including embedded servers)
+    this.segmentDeepStorageDirectory = new MountedDir(
+        "/tmp/druid/deep-store",
+        cluster.getTestFolder().getOrCreateFolder("deep-store").getAbsolutePath()
+    );
+    this.indexerLogsDeepStorageDirectory = new MountedDir(
+        "/tmp/druid/indexer-logs",
+        cluster.getTestFolder().getOrCreateFolder("indexer-logs").getAbsolutePath()
+    );
+
+    // Mount directories used by this container for easier debugging with service logs
+    this.containerDirectory = cluster.getTestFolder().getOrCreateFolder(name).getAbsolutePath();
+    this.serviceLogsDirectory = new MountedDir(
+        "/opt/druid/log",
+        containerDirectory + "/log"
+    );
+
+    // Create the log directory upfront to avoid permission issues
+    createLogDirectory(new File(containerDirectory, "log"));
+  }
+
+  @Override
+  protected DruidContainer createContainer()
+  {
+    final List<Integer> exposedPorts = new ArrayList<>(taskPorts);
+    exposedPorts.add(servicePort);
+    log.info(
+        "Starting Druid container[%s] on port[%d] with mounted directory[%s] and exposed ports[%s].",
+        name, servicePort, containerDirectory, exposedPorts
+    );
+
+    final DruidContainer container = new DruidContainer(command, imageName)
+        .withExposedPorts(exposedPorts.toArray(new Integer[0]))
+        .withCommonProperties(getCommonProperties())
+        .withServiceProperties(getServerProperties())
+        .withFileSystemBind(
+            segmentDeepStorageDirectory.hostPath,
+            segmentDeepStorageDirectory.containerPath,
+            BindMode.READ_WRITE
+        )

Review Comment:
   added a new method.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.

kfaraz commented on code in PR #18302:
URL: https://github.com/apache/druid/pull/18302#discussion_r2246925782


##########
.github/workflows/docker-tests.yml:
##########
@@ -0,0 +1,97 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: "Docker Tests using Distribution Image"
+on:
+  workflow_call:
+
+jobs:
+  run-docker-tests:
+    name: Run Docker tests
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Retrieve Docker container
+        id: docker-restore
+        uses: actions/cache/restore@v4
+        with:
+          key: druid-dist-container.tar.gz-${{ github.sha }}
+          path: |
+            ./druid-dist-container.tar.gz
+      - name: Stop and remove Druid Docker containers
+        run: |
+          echo "Force stopping all Druid containers and pruning"
+          docker ps -aq --filter "ancestor=apache/druid" | xargs -r docker rm -f
+          docker system prune -af --volumes
+      - name: Load Docker image
+        run: |
+          docker load --input druid-dist-container.tar.gz
+          docker images
+      - name: Setup Java
+        uses: actions/setup-java@v4
+        with:
+          distribution: 'zulu'
+          java-version: 17
+          cache: 'maven'
+      - name: Run *DockerTest*
+        id: run-it
+        run: |
+          source .github/scripts/distribution_checks_env.sh
+          .github/scripts/run_unit-tests -Dtest=*DockerTest* -Ddruid.testing.docker.image=$DRUID_DIST_IMAGE_NAME

Review Comment:
   Added script `run_docker-tests`. To run all Docker tests locally,
   
   ```
   $ DRUID_DIST_IMAGE_NAME=apache/druid:34.0.0-rc1 .github/scripts/run_docker-tests
   ```
   
   This doesn't include the image building yet though.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.

kfaraz commented on code in PR #18302:
URL: https://github.com/apache/druid/pull/18302#discussion_r2238958383


##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/IngestionDockerTest.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.druid.common.utils.IdUtils;
+import org.apache.druid.data.input.impl.CsvInputFormat;
+import org.apache.druid.data.input.impl.DimensionsSpec;
+import org.apache.druid.data.input.impl.TimestampSpec;
+import org.apache.druid.indexer.TaskState;
+import org.apache.druid.indexer.TaskStatusPlus;
+import org.apache.druid.indexing.kafka.KafkaIndexTaskModule;
+import org.apache.druid.indexing.kafka.simulate.KafkaResource;
+import org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorIOConfig;
+import org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorSpec;
+import org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorTuningConfig;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.supervisor.SupervisorStatus;
+import org.apache.druid.java.util.common.DateTimes;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.parsers.CloseableIterator;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.indexing.DataSchema;
+import org.apache.druid.testing.embedded.EmbeddedBroker;
+import org.apache.druid.testing.embedded.EmbeddedClusterApis;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.EmbeddedOverlord;
+import org.apache.druid.testing.embedded.EmbeddedRouter;
+import org.apache.druid.testing.embedded.derby.EmbeddedDerbyMetadataResource;
+import org.apache.druid.testing.embedded.indexing.Resources;
+import org.apache.druid.testing.embedded.junit5.EmbeddedClusterTestBase;
+import org.apache.druid.testing.embedded.minio.MinIOStorageResource;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.joda.time.DateTime;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ThreadLocalRandom;
+
+/**
+ * Runs some basic ingestion tests using {@link DruidContainers}.
+ */
+public class IngestionDockerTest extends EmbeddedClusterTestBase
+{
+  static {
+    System.setProperty("druid.testing.docker.image", "apache/druid:35.0.0-SNAPSHOT");

Review Comment:
   For local testing, what do you think would be the best way to specify this property?
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "github-advanced-security[bot] (via GitHub)" <gi...@apache.org>.

github-advanced-security[bot] commented on code in PR #18302:
URL: https://github.com/apache/druid/pull/18302#discussion_r2239133498


##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/DruidContainerResource.java:
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import org.apache.druid.java.util.common.FileUtils;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.testing.DruidCommand;
+import org.apache.druid.testing.DruidContainer;
+import org.apache.druid.testing.MountedDir;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.TestcontainerResource;
+import org.testcontainers.utility.DockerImageName;
+
+import java.io.File;
+import java.nio.file.Files;
+import java.nio.file.attribute.PosixFilePermissions;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * {@link TestcontainerResource} to run Druid services.
+ * Currently, only core extensions can be used out-of-the-box with these containers
+ * such as {@code druid-s3-extensions} or {@code postgresql-metadata-storage},
+ * simply by adding them to {@code druid.extensions.loadList}.
+ * <p>
+ * {@link DruidContainers} should be used only for testing backward compatiblity
+ * or a Docker-specific feature. For all other testing needs, use plain old
+ * {@code EmbeddedDruidServer} as they are much faster, allow easy debugging and
+ * do not require downloading any images.
+ */
+public class DruidContainerResource extends TestcontainerResource<DruidContainer>
+{
+  /**
+   * Java system property to specify the name of the Docker test image.
+   */
+  public static final String PROPERTY_TEST_IMAGE = "druid.testing.docker.image";
+
+  private static final Logger log = new Logger(DruidContainerResource.class);
+
+  /**
+   * Forbidden server properties that may be used by EmbeddedDruidServers but
+   * interfere with the functioning of DruidContainer-based services.
+   */
+  private static final Set<String> FORBIDDEN_PROPERTIES = Set.of(
+      "druid.extensions.modulesForEmbeddedTests",
+      "druid.emitter"
+  );
+
+  /**
+   * A static incremental ID is used instead of a random number to ensure that
+   * tests are more deterministic and easier to debug.
+   */
+  private static final AtomicInteger SERVER_ID = new AtomicInteger(0);
+
+  private final String name;
+  private final DruidCommand command;
+  private final Map<String, String> properties = new HashMap<>();
+
+  private DockerImageName imageName;
+  private EmbeddedDruidCluster cluster;
+
+  private File containerDirectory;
+
+  private MountedDir indexerLogsDeepStorageDirectory;
+  private MountedDir serviceLogsDirectory;
+  private MountedDir segmentDeepStorageDirectory;
+
+  DruidContainerResource(DruidCommand command)
+  {
+    this.name = StringUtils.format(
+        "container_%s_%d",
+        command.getName(),
+        SERVER_ID.incrementAndGet()
+    );
+    this.command = command;
+    addProperty("druid.host", EmbeddedDruidCluster.getDefaultHost());
+  }
+
+  public DruidContainerResource usingImage(DockerImageName imageName)
+  {
+    this.imageName = imageName;
+    return this;
+  }
+
+  /**
+   * Uses the Docker test image specified by the system property
+   * {@link #PROPERTY_TEST_IMAGE} for this container.
+   */
+  public DruidContainerResource usingTestImage()
+  {
+    String imageName = Objects.requireNonNull(
+        System.getProperty(PROPERTY_TEST_IMAGE),
+        StringUtils.format("System property[%s] is not set", PROPERTY_TEST_IMAGE)
+    );
+    return usingImage(DockerImageName.parse(imageName));
+  }
+
+  public DruidContainerResource addProperty(String key, String value)
+  {
+    properties.put(key, value);
+    return this;
+  }
+
+  @Override
+  public void beforeStart(EmbeddedDruidCluster cluster)
+  {
+    this.cluster = cluster;
+
+    // Mount directories used by the entire cluster (including embedded servers)
+    this.segmentDeepStorageDirectory = new MountedDir(
+        new File("/druid/deep-store"),
+        cluster.getTestFolder().getOrCreateFolder("deep-store")
+    );
+    this.indexerLogsDeepStorageDirectory = new MountedDir(
+        new File("/druid/indexer-logs"),
+        cluster.getTestFolder().getOrCreateFolder("indexer-logs")
+    );
+
+    // Mount directories used by this container for easier debugging with service logs
+    this.containerDirectory = cluster.getTestFolder().getOrCreateFolder(name);
+
+    final File logDirectory = new File(containerDirectory, "log");
+    this.serviceLogsDirectory = new MountedDir(new File("/opt/druid/log"), logDirectory);
+
+    // Create the log directory upfront to avoid permission issues
+    createLogDirectory(logDirectory);
+  }
+
+  @Override
+  protected DruidContainer createContainer()
+  {
+    final DruidContainer container = new DruidContainer(command, imageName)
+        .withFileSystemBind(segmentDeepStorageDirectory)
+        .withFileSystemBind(indexerLogsDeepStorageDirectory)
+        .withFileSystemBind(serviceLogsDirectory)
+        .withEnv(
+            Map.of(
+                "DRUID_SET_HOST_IP", "0",
+                "DRUID_SET_HOST", "0"
+            )
+        );
+
+    log.info(
+        "Starting Druid container[%s] with mounted directory[%s] and exposed ports[%s].",
+        name, containerDirectory, command.getExposedPorts()

Review Comment:
   ## Implicit conversion from array to string
   
   Implicit conversion from Array to String.
   
   [Show more details](https://github.com/apache/druid/security/code-scanning/9931)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.

kfaraz commented on code in PR #18302:
URL: https://github.com/apache/druid/pull/18302#discussion_r2239393498


##########
services/src/test/java/org/apache/druid/testing/embedded/EmbeddedDruidCluster.java:
##########
@@ -231,13 +242,50 @@ public TestFolder getTestFolder()
   }
 
   /**
-   * The embedded Zookeeper server used by this cluster, if any.
-   *
-   * @throws NullPointerException if this cluster has no embedded zookeeper.
+   * Hostname to be used for embedded services (both Druid or external).
+   * Using this hostname ensures that the underlying service is reachable by both
+   * EmbeddedDruidServers and DruidContainers.
+   */
+  public String getEmbeddedServiceHostname()
+  {
+    return hasDruidContainers ? getDefaultHost() : "localhost";
+  }
+
+  /**
+   * Hostname for the host machine running the containers. Using this hostname
+   * instead of "localhost" allows all the Druid containers to talk to each
+   * other and also other EmbeddedDruidServers.
    */
-  public EmbeddedZookeeper getZookeeper()
+  public static String getDefaultHost()
   {
-    return Objects.requireNonNull(zookeeper, "No embedded zookeeper configured for this cluster");
+    try {
+      return InetAddress.getLocalHost().getHostAddress();
+    }
+    catch (UnknownHostException e) {
+      throw new ISE(e, "Unable to determine host name");
+    }
+  }
+
+  /**
+   * Replaces {@code localhost} or {@code 127.0.0.1} in the given connectUri
+   * with {@link #getEmbeddedServiceHostname()}. Using the embedded URI ensures
+   * that the underlying service is reachable by both EmbeddedDruidServers and
+   * DruidContainers.
+   */
+  public String getEmbeddedConnectUri(String connectUri)
+  {
+    if (!hasDruidContainers) {
+      return connectUri;
+    } else if (connectUri.contains("localhost")) {
+      return StringUtils.replace(connectUri, "localhost", getEmbeddedServiceHostname());

Review Comment:
   fair enough 😂 , I have updated it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.

kfaraz commented on code in PR #18302:
URL: https://github.com/apache/druid/pull/18302#discussion_r2255779892


##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/IngestionBackwardCompatibilityDockerTest.java:
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.druid.client.coordinator.CoordinatorServiceClient;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.http.client.response.BytesFullResponseHandler;
+import org.apache.druid.java.util.http.client.response.BytesFullResponseHolder;
+import org.apache.druid.rpc.RequestBuilder;
+import org.apache.druid.rpc.indexing.SegmentUpdateResponse;
+import org.apache.druid.testing.DruidContainer;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.jboss.netty.handler.codec.http.HttpMethod;
+import org.junit.jupiter.api.Nested;
+
+/**
+ * Runs some basic ingestion tests using {@code DruidContainers} to verify
+ * backward compatibility with old Druid images.
+ */
+public class IngestionBackwardCompatibilityDockerTest
+{
+  @Nested
+  public class Apache31 extends IngestionDockerTest
+  {
+    @Override
+    public EmbeddedDruidCluster createCluster()
+    {
+      coordinator.usingImage(DruidContainer.Image.APACHE_31);

Review Comment:
   No, I think we can keep testing against 31 until we decide to make some major change which is not backward compatible with 31 anymore.
   
   I didn't try using an image older than 31, but we could probably go as far back as 27 and still be backward compatible (at least for the things that are being verified in these tests).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "capistrant (via GitHub)" <gi...@apache.org>.

capistrant commented on code in PR #18302:
URL: https://github.com/apache/druid/pull/18302#discussion_r2257543326


##########
services/src/test/java/org/apache/druid/testing/embedded/EmbeddedDruidCluster.java:
##########
@@ -232,13 +228,21 @@ public TestFolder getTestFolder()
   }
 
   /**
-   * The embedded Zookeeper server used by this cluster, if any.
-   *
-   * @throws NullPointerException if this cluster has no embedded zookeeper.
+   * Uses a container-friendly hostname for all embedded services, Druid as well
+   * as external.
+   */
+  public EmbeddedDruidCluster useContainerFriendlyHostname()

Review Comment:
   From an ease of use perspective I think it would be awesome to get to a point where we default to container friendly so people who are writing new tests don't need to have to decide/realize their test requires it. But I don't think it is blocking for this PR



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add DruidContainers to run docker tests with embedded-test framework (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.

kfaraz commented on code in PR #18302:
URL: https://github.com/apache/druid/pull/18302#discussion_r2259594279


##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/IngestionBackwardCompatibilityDockerTest.java:
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.docker;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.druid.client.coordinator.CoordinatorServiceClient;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.http.client.response.BytesFullResponseHandler;
+import org.apache.druid.java.util.http.client.response.BytesFullResponseHolder;
+import org.apache.druid.rpc.RequestBuilder;
+import org.apache.druid.rpc.indexing.SegmentUpdateResponse;
+import org.apache.druid.testing.DruidContainer;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.jboss.netty.handler.codec.http.HttpMethod;
+import org.junit.jupiter.api.Nested;
+
+/**
+ * Runs some basic ingestion tests using {@code DruidContainers} to verify
+ * backward compatibility with old Druid images.
+ */
+public class IngestionBackwardCompatibilityDockerTest
+{
+  @Nested
+  public class Apache31 extends IngestionDockerTest
+  {
+    @Override
+    public EmbeddedDruidCluster createCluster()
+    {
+      coordinator.usingImage(DruidContainer.Image.APACHE_31);

Review Comment:
   > Btw, IMO it's nicer to use annotations rather than test names to control what tests run in a standard mvn run. The downside of using names is that it's indirect. It makes it less obvious which tests run and which don't, and also means that if you want to change whether a test runs, you have to rename it.
   
   I agree. In the current code, I am using the `@Tag("docker-tests")` annotation (applied on `DockerTestBase`) to skip the Docker-based tests.
   The `*DockerTest*` class name is to tell `mvn verify` to look for these classes while searching for eligible integration tests. Otherwise, it only looks for `*IT` or `IT*` classes by default.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org