You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2026/09/02 16:44:49 UTC
(spark-connect-swift) branch main updated: [SPARK-59177] Use lower-bound version checks in `SQLTests` and `SparkConnectClientTests`
This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-connect-swift.git
The following commit(s) were added to refs/heads/main by this push:
new bc83c44 [SPARK-59177] Use lower-bound version checks in `SQLTests` and `SparkConnectClientTests`
bc83c44 is described below
commit bc83c441fe7af8b542b4ec4d183dcea520f65594
Author: Dongjoon Hyun <do...@apache.org>
AuthorDate: Wed Sep 2 09:44:42 2026 -0700
[SPARK-59177] Use lower-bound version checks in `SQLTests` and `SparkConnectClientTests`
### What changes were proposed in this pull request?
This PR aims to replace the exact version-prefix checks in the test suites with lower-bound version comparisons, so that the version-gated tests also run on newer Apache Spark servers.
- `SQLTests`: `queriesForSpark42Only` is renamed to `queriesForSpark42AndLater` and the skip condition becomes `spark.version < "4.2"` instead of `!spark.version.starts(with: "4.2")`.
- `SparkConnectClientTests`: the five Declarative Pipelines API gates become `response.sparkVersion.version >= "4.1"` instead of `response.sparkVersion.version.starts(with: "4.1")`.
### Why are the changes needed?
The existing conditions match a single feature version only, so the tests are silently skipped on any later server.
| Test | Spark 4.0 | Spark 4.1 | Spark 4.2 | Spark 4.3 |
| --- | --- | --- | --- | --- |
| `SQLTests`: `geography.sql`, `geometry.sql`, `time.sql` | skipped | skipped | run | **skipped** |
| `SparkConnectClientTests`: `createDataflowGraph`, `defineSqlGraphElements` | skipped | run | **skipped** | **skipped** |
In other words, the newly added `integration-test-mac-spark43` job (SPARK-59156) loses the `TIME`/geospatial SQL coverage, and the Declarative Pipelines APIs have not been covered by any job since Apache Spark 4.2.0. Since these features are available in all later versions, the gates should be lower bounds. This also matches the style already used across the test suites, e.g. `if await spark.version >= "4.2"`.
### Does this PR introduce _any_ user-facing change?
No. This is a test-only change.
### How was this patch tested?
Pass the CIs.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5
Closes #501 from dongjoon-hyun/SPARK-59177.
Authored-by: Dongjoon Hyun <do...@apache.org>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
Tests/SparkConnectTests/SQLTests.swift | 7 ++++---
Tests/SparkConnectTests/SparkConnectClientTests.swift | 10 +++++-----
2 files changed, 9 insertions(+), 8 deletions(-)
diff --git a/Tests/SparkConnectTests/SQLTests.swift b/Tests/SparkConnectTests/SQLTests.swift
index 0677c6d..bd83c38 100644
--- a/Tests/SparkConnectTests/SQLTests.swift
+++ b/Tests/SparkConnectTests/SQLTests.swift
@@ -90,7 +90,8 @@ struct SQLTests {
#expect(normalize("+ + +") == "+ + +")
}
- let queriesForSpark42Only: [String] = [
+ /// Queries which are supported by Apache Spark 4.2.0 and later.
+ let queriesForSpark42AndLater: [String] = [
"geography.sql",
"geometry.sql",
"time.sql",
@@ -103,8 +104,8 @@ struct SQLTests {
for name in try! fm.contentsOfDirectory(atPath: path).sorted() {
guard name.hasSuffix(".sql") else { continue }
print(name)
- if await !spark.version.starts(with: "4.2") && queriesForSpark42Only.contains(name) {
- print("Skip query \(name) due to the difference between Spark 4.1 and 4.2")
+ if await spark.version < "4.2" && queriesForSpark42AndLater.contains(name) {
+ print("Skip query \(name) which requires Spark 4.2 or later")
continue
}
diff --git a/Tests/SparkConnectTests/SparkConnectClientTests.swift b/Tests/SparkConnectTests/SparkConnectClientTests.swift
index 3b11777..302c9c4 100644
--- a/Tests/SparkConnectTests/SparkConnectClientTests.swift
+++ b/Tests/SparkConnectTests/SparkConnectClientTests.swift
@@ -132,7 +132,7 @@ struct SparkConnectClientTests {
func createDataflowGraph() async throws {
let client = try SparkConnectClient(remote: TEST_REMOTE)
let response = try await client.connect(UUID().uuidString)
- if response.sparkVersion.version.starts(with: "4.1") {
+ if response.sparkVersion.version >= "4.1" {
let dataflowGraphID = try await client.createDataflowGraph()
#expect(UUID(uuidString: dataflowGraphID) != nil)
}
@@ -148,7 +148,7 @@ struct SparkConnectClientTests {
try await client.startRun("not-a-uuid-format")
}
- if response.sparkVersion.version.starts(with: "4.1") {
+ if response.sparkVersion.version >= "4.1" {
let dataflowGraphID = try await client.createDataflowGraph()
#expect(UUID(uuidString: dataflowGraphID) != nil)
#expect(try await client.startRun(dataflowGraphID))
@@ -165,7 +165,7 @@ struct SparkConnectClientTests {
try await client.defineOutput("not-a-uuid-format", "ds1", "table")
}
- if response.sparkVersion.version.starts(with: "4.1") {
+ if response.sparkVersion.version >= "4.1" {
let dataflowGraphID = try await client.createDataflowGraph()
#expect(UUID(uuidString: dataflowGraphID) != nil)
try await #require(throws: SparkConnectError.OutputTypeUnspecified) {
@@ -200,7 +200,7 @@ struct SparkConnectClientTests {
try await client.defineFlow("not-a-uuid-format", "f1", "ds1", Relation())
}
- if response.sparkVersion.version.starts(with: "4.1") {
+ if response.sparkVersion.version >= "4.1" {
let dataflowGraphID = try await client.createDataflowGraph()
#expect(UUID(uuidString: dataflowGraphID) != nil)
let relation = await client.getLocalRelation().root
@@ -218,7 +218,7 @@ struct SparkConnectClientTests {
try await client.defineSqlGraphElements("not-a-uuid-format", "path", "sql")
}
- if response.sparkVersion.version.starts(with: "4.1") {
+ if response.sparkVersion.version >= "4.1" {
let dataflowGraphID = try await client.createDataflowGraph()
let sqlText = "CREATE MATERIALIZED VIEW mv1 AS SELECT 1"
#expect(UUID(uuidString: dataflowGraphID) != nil)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org