You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2024/01/22 06:37:42 UTC
(spark) branch master updated: [SPARK-46788][DOCS] Add instructions for running Docker integration tests
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new aa4944f99566 [SPARK-46788][DOCS] Add instructions for running Docker integration tests
aa4944f99566 is described below
commit aa4944f995664d8d8d61a085c8645f138712e675
Author: Kent Yao <ya...@apache.org>
AuthorDate: Sun Jan 21 22:37:27 2024 -0800
[SPARK-46788][DOCS] Add instructions for running Docker integration tests
### What changes were proposed in this pull request?
Following the review comment https://github.com/apache/spark/pull/44612#pullrequestreview-1827546783, in this PR, we add a README for instructions on Docker integration tests
### Why are the changes needed?
tips for Docker integration tests with different envrionments
### Does this PR introduce _any_ user-facing change?
no
### How was this patch tested?
doc build
### Was this patch authored or co-authored using generative AI tooling?
no
Closes #44826 from yaooqinn/SPARK-46788.
Authored-by: Kent Yao <ya...@apache.org>
Signed-off-by: Dongjoon Hyun <dh...@apple.com>
---
connector/docker-integration-tests/README.md | 111 +++++++++++++++++++++++++++
1 file changed, 111 insertions(+)
diff --git a/connector/docker-integration-tests/README.md b/connector/docker-integration-tests/README.md
new file mode 100644
index 000000000000..61501be91ca2
--- /dev/null
+++ b/connector/docker-integration-tests/README.md
@@ -0,0 +1,111 @@
+---
+layout: global
+title: Spark Docker Integration Tests
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+# Running the Docker integration tests
+
+Note that the integration test framework is using thirdparty Docker images to test functionalities
+of integration with other data sources, not running Spark itself in Docker containers, which is
+actually done by the Kubernetes Integration Tests module.
+
+In order to run the Docker integration tests, the [Docker engine](https://docs.docker.com/engine/installation/)
+needs to be installed and started on the machine. Additionally, the environment variable
+`ENABLE_DOCKER_INTEGRATION_TESTS=1` shall be specified to enable the Docker integration tests.
+
+ export ENABLE_DOCKER_INTEGRATION_TESTS=1
+
+or,
+
+ ENABLE_DOCKER_INTEGRATION_TESTS=1 ./build/sbt -Pdocker-integration-tests "docker-integration-tests/test"
+
+## Running an individual Docker integration test
+
+Testing the whole module of Docker Integration Tests might be time-consuming because of image pulling and
+the container bootstrapping. To run an individual Docker integration test, use the following command:
+
+ ./build/sbt -Pdocker-integration-tests "docker-integration-tests/testOnly <test class name>"
+
+## Using a custom Docker image
+
+Besides the default Docker images, the integration tests can be run with custom Docker images. For example,
+
+ ORACLE_DOCKER_IMAGE_NAME=gvenzl/oracle-free:23.3-slim-faststart ./build/sbt -Pdocker-integration-tests "docker-integration-tests/testOnly *OracleIntegrationSuite"
+
+The following environment variables can be used to specify the custom Docker images for different databases:
+
+- DB2_DOCKER_IMAGE_NAME
+- MARIADB_DOCKER_IMAGE_NAME
+- MSSQLSERVER_DOCKER_IMAGE_NAME
+- MYSQL_DOCKER_IMAGE_NAME
+- ORACLE_DOCKER_IMAGE_NAME
+- POSTGRES_DOCKER_IMAGE_NAME
+
+## Using a custom Docker context
+
+In certain scenarios, you may want to use a different docker context/endpoint instead of the default provided by
+Docker Desktop.
+
+ docker context ls
+ NAME TYPE DESCRIPTION DOCKER ENDPOINT KUBERNETES ENDPOINT ORCHESTRATOR
+ default * moby Current DOCKER_HOST based configuration unix:///var/run/docker.sock
+ desktop-linux moby Docker Desktop unix:///Users/.../docker.sock
+
+ docker context use desktop-linux
+
+Then you can run the integration tests as usual targeting the Docker endpoint named `desktop-linux`.
+
+This is useful when the upstream Docker image can not starton the Docker Desktop. For example, when the image you
+use only supports `x86_64` architecture, but you are running on an Apple Silicon `aarch64` machine, for which case
+you may want to use a custom context that is able to mock a `x86_64` architecture, such as `colima`. After
+[colima](https://github.com/abiosoft/colima) installed, you can start a runtime with `x86_64` support and run the
+integration tests as follows:
+
+ colima start --arch x86_64 --memory 8 --network-address
+ docker context colima
+ ./build/sbt -Pdocker-integration-tests "docker-integration-tests/testOnly *OracleIntegrationSuite"
+
+## Available Properties
+
+The following are the available properties that can be passed to optimize testing experience.
+
+ ./build/sbt -Pdocker-integration-tests \
+ -Dspark.test.docker.keepContainer=true \
+ "testOnly *MariaDBKrbIntegrationSuite"
+
+<table>
+ <tr>
+ <th>Property</th>
+ <th>Description</th>
+ <th>Default</th>
+ </tr>
+ <tr>
+ <td><code>spark.test.docker.keepContainer</code></td>
+ <td>
+ When true, the Docker container used for the test will keep running after all tests finished in a single test file.
+ </td>
+ <td>false</td>
+ </tr>
+ <tr>
+ <td><code>spark.test.docker.removePulledImage</code></td>
+ <td>
+ When true, the Docker image used for the test will not be preserved after all tests finished in a single test file.
+ </td>
+ <td>true</td>
+ </tr>
+</table>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org