You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ya...@apache.org on 2020/11/28 14:45:52 UTC
[spark] branch master updated: [SPARK-33570][SQL][TESTS] Set the
proper version of gssapi plugin automatically for
MariaDBKrbIntegrationSuite
This is an automated email from the ASF dual-hosted git repository.
yamamuro pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new cf98a76 [SPARK-33570][SQL][TESTS] Set the proper version of gssapi plugin automatically for MariaDBKrbIntegrationSuite
cf98a76 is described below
commit cf98a761de677c733f3c33230e1c63ddb785d5c5
Author: Kousuke Saruta <sa...@oss.nttdata.com>
AuthorDate: Sat Nov 28 23:38:11 2020 +0900
[SPARK-33570][SQL][TESTS] Set the proper version of gssapi plugin automatically for MariaDBKrbIntegrationSuite
### What changes were proposed in this pull request?
This PR changes mariadb_docker_entrypoint.sh to set the proper version automatically for mariadb-plugin-gssapi-server.
The proper version is based on the one of mariadb-server.
Also, this PR enables to use arbitrary docker image by setting the environment variable `MARIADB_CONTAINER_IMAGE_NAME`.
### Why are the changes needed?
For `MariaDBKrbIntegrationSuite`, the version of `mariadb-plugin-gssapi-server` is currently set to `10.5.5` in `mariadb_docker_entrypoint.sh` but it's no longer available in the official apt repository and `MariaDBKrbIntegrationSuite` doesn't pass for now.
It seems that only the most recent three versions are available for each major version and they are `10.5.6`, `10.5.7` and `10.5.8` for now.
Further, the release cycle of MariaDB seems to be very rapid (1 ~ 2 months) so I don't think it's a good idea to set to an specific version for `mariadb-plugin-gssapi-server`.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Confirmed that `MariaDBKrbIntegrationSuite` passes with the following commands.
```
$ build/sbt -Pdocker-integration-tests -Phive -Phive-thriftserver package "testOnly org.apache.spark.sql.jdbc.MariaDBKrbIntegrationSuite"
```
In this case, we can see what version of `mariadb-plugin-gssapi-server` is going to be installed in the following container log message.
```
Installing mariadb-plugin-gssapi-server=1:10.5.8+maria~focal
```
Or, we can set MARIADB_CONTAINER_IMAGE_NAME for a specific version of MariaDB.
```
$ MARIADB_DOCKER_IMAGE_NAME=mariadb:10.5.6 build/sbt -Pdocker-integration-tests -Phive -Phive-thriftserver package "testOnly org.apache.spark.sql.jdbc.MariaDBKrbIntegrationSuite"
```
```
Installing mariadb-plugin-gssapi-server=1:10.5.6+maria~focal
```
Closes #30515 from sarutak/fix-MariaDBKrbIntegrationSuite.
Authored-by: Kousuke Saruta <sa...@oss.nttdata.com>
Signed-off-by: Takeshi Yamamuro <ya...@apache.org>
---
.../src/test/resources/mariadb_docker_entrypoint.sh | 4 +++-
.../apache/spark/sql/jdbc/MariaDBKrbIntegrationSuite.scala | 12 +++++++++---
2 files changed, 12 insertions(+), 4 deletions(-)
diff --git a/external/docker-integration-tests/src/test/resources/mariadb_docker_entrypoint.sh b/external/docker-integration-tests/src/test/resources/mariadb_docker_entrypoint.sh
index 97c00a9..ab7d967 100755
--- a/external/docker-integration-tests/src/test/resources/mariadb_docker_entrypoint.sh
+++ b/external/docker-integration-tests/src/test/resources/mariadb_docker_entrypoint.sh
@@ -18,7 +18,9 @@
dpkg-divert --add /bin/systemctl && ln -sT /bin/true /bin/systemctl
apt update
-apt install -y mariadb-plugin-gssapi-server=1:10.5.5+maria~focal
+GSSAPI_PLUGIN=mariadb-plugin-gssapi-server=$(dpkg -s mariadb-server | sed -n "s/^Version: \(.*\)/\1/p")
+echo "Installing $GSSAPI_PLUGIN"
+apt install -y "$GSSAPI_PLUGIN"
echo "gssapi_keytab_path=/docker-entrypoint-initdb.d/mariadb.keytab" >> /etc/mysql/mariadb.conf.d/auth_gssapi.cnf
echo "gssapi_principal_name=mariadb/__IP_ADDRESS_REPLACE_ME__@EXAMPLE.COM" >> /etc/mysql/mariadb.conf.d/auth_gssapi.cnf
docker-entrypoint.sh mysqld
diff --git a/external/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/MariaDBKrbIntegrationSuite.scala b/external/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/MariaDBKrbIntegrationSuite.scala
index adee2be..59a6f53 100644
--- a/external/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/MariaDBKrbIntegrationSuite.scala
+++ b/external/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/MariaDBKrbIntegrationSuite.scala
@@ -24,15 +24,21 @@ import com.spotify.docker.client.messages.{ContainerConfig, HostConfig}
import org.apache.spark.sql.execution.datasources.jdbc.connection.SecureConnectionProvider
import org.apache.spark.tags.DockerTest
+/**
+ * To run this test suite for a specific version (e.g., mariadb:10.5.8):
+ * {{{
+ * MARIADB_DOCKER_IMAGE_NAME=mariadb:10.5.8
+ * ./build/sbt -Pdocker-integration-tests
+ * "testOnly org.apache.spark.sql.jdbc.MariaDBKrbIntegrationSuite"
+ * }}}
+ */
@DockerTest
class MariaDBKrbIntegrationSuite extends DockerKrbJDBCIntegrationSuite {
override protected val userName = s"mariadb/$dockerIp"
override protected val keytabFileName = "mariadb.keytab"
override val db = new DatabaseOnDocker {
- // If you change `imageName`, you need to update the version of `mariadb-plugin-gssapi-server`
- // in `resources/mariadb_docker_entrypoint.sh` accordingly.
- override val imageName = "mariadb:10.5"
+ override val imageName = sys.env.getOrElse("MARIADB_DOCKER_IMAGE_NAME", "mariadb:10.5")
override val env = Map(
"MYSQL_ROOT_PASSWORD" -> "rootpass"
)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org