You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2026/08/28 07:49:25 UTC
(spark-connect-rust) branch master updated: [SPARK-59073][INFRA] Run resource & artifact tests against a local-cluster Connect server
This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new bda9f55 [SPARK-59073][INFRA] Run resource & artifact tests against a local-cluster Connect server
bda9f55 is described below
commit bda9f550e21a824a1c03f0933bdc32db0c3a8066
Author: Hyukjin Kwon <hy...@databricks.com>
AuthorDate: Fri Aug 28 16:49:17 2026 +0900
[SPARK-59073][INFRA] Run resource & artifact tests against a local-cluster Connect server
### What changes were proposed in this pull request?
Mirror Apache Spark's `build_python_connect.yml`, which runs the resource-profile and artifact tests against a **local-cluster** master (a real multi-executor cluster) rather than `local[*]`. The single-JVM local mode cannot exercise resource profiles or artifact distribution across executors, so those tests were previously either not meaningful (run against `local[*]`) or not run at all (`test_connect_resources` lives outside the discovered `sql/tests/connect` scope).
- `run_official_tests.py` gains:
- `--only`: run an explicit list of tests, given either as dotted module names (like Apache's `run-tests --testnames`, e.g. `pyspark.resource.tests.test_connect_resources`) or `spark_py`-relative paths. This can reach files outside the `sql/tests/connect` scope `discover()` walks.
- `--exclude`: drop file basenames from discovery.
- The parity job bounces the Connect server into `local-cluster[2, 4, 1024]` after the main `local[*]` run and runs the same four tests Apache does, through our Rust client:
- `pyspark.resource.tests.test_connect_resources`
- `pyspark.sql.tests.connect.client.test_artifact`
- `pyspark.sql.tests.connect.client.test_artifact_localcluster`
- `pyspark.sql.tests.connect.test_parity_resources`
- The three that live under `sql/tests/connect` are excluded from the `local[*]` run so they execute only where they are meaningful (matching Apache, which runs them only under local-cluster).
### Why are the changes needed?
Resource profiles and cross-executor artifact distribution are untested against a single-JVM server. This adds the local-cluster coverage Apache already has.
### Does this PR introduce _any_ user-facing change?
No. CI-only.
### How was this patch tested?
CI (this PR). `run_official_tests.py --only` resolution verified locally; the local-cluster phase runs in CI.
This pull request and its description were written by Isaac.
Closes #74 from HyukjinKwon/feature/local-cluster-resource-artifact.
Authored-by: Hyukjin Kwon <hy...@databricks.com>
Signed-off-by: Hyukjin Kwon <hy...@databricks.com>
---
.github/workflows/build_python_connect.yml | 62 ++++++++++++++++++++++++++++++
scripts/run_official_tests.py | 51 +++++++++++++++++++++++-
2 files changed, 112 insertions(+), 1 deletion(-)
diff --git a/.github/workflows/build_python_connect.yml b/.github/workflows/build_python_connect.yml
index 7117ba7..0296120 100644
--- a/.github/workflows/build_python_connect.yml
+++ b/.github/workflows/build_python_connect.yml
@@ -199,12 +199,72 @@ jobs:
# (./python/run-tests --parallelism=1): the streaming-query-listener tests wait
# on real server-pushed progress events and are timeout-sensitive, so parallel
# runs starve them under server contention. Serial removes that artifact.
+ # The resource/artifact tests need a real multi-executor cluster, so they run
+ # in the separate local-cluster phase below (mirroring Apache's
+ # build_python_connect.yml, which runs exactly these under local-cluster) and
+ # are excluded here where the server is plain local[*].
python scripts/run_official_tests.py \
--spark "${{ steps.spark-source.outputs.SPARK_SOURCE }}" \
--remote "${SPARK_CONNECT_TESTING_REMOTE}" \
--jobs 1 \
+ --exclude "test_artifact.py,test_artifact_localcluster.py,test_parity_resources.py" \
2>&1 | tee /tmp/parity_ours.log
+ - name: Restart Connect server in local-cluster mode
+ # Resource profiles and artifact distribution across executors can only be
+ # exercised against a real multi-executor cluster, so - mirroring Apache's
+ # build_python_connect.yml - bounce the server into local-cluster mode for those
+ # tests (the parity run above uses the default local[*] and excludes these files).
+ env:
+ PYSPARK_PYTHON: python
+ PYSPARK_DRIVER_PYTHON: python
+ run: |
+ SPARK_HOME="${{ steps.spark-download.outputs.SPARK_HOME }}"
+ echo "Stopping the local[*] Connect server..."
+ "${SPARK_HOME}/sbin/stop-connect-server.sh" || true
+ pkill -f 'org.apache.spark.sql.connect.service' 2>/dev/null || true
+ pkill -f 'spark-connect' 2>/dev/null || true
+ for i in $(seq 1 30); do
+ if ! (echo > /dev/tcp/localhost/15002) 2>/dev/null; then break; fi
+ sleep 1
+ done
+ echo "Starting Connect server in local-cluster mode..."
+ # Keep --packages: local-cluster launches real executor JVMs that each need the
+ # spark-connect jar on their classpath, so it is not redundant here.
+ "${SPARK_HOME}/sbin/start-connect-server.sh" \
+ --master "local-cluster[2, 4, 1024]" \
+ --packages "org.apache.spark:spark-connect_2.13:${SPARK_VERSION}" \
+ --driver-memory 3g \
+ --conf spark.sql.session.localRelationSizeLimit=1073741824 \
+ > /tmp/connect-server-localcluster.log 2>&1 &
+ for i in $(seq 1 120); do
+ if (echo > /dev/tcp/localhost/15002) 2>/dev/null; then
+ echo "Connect server (local-cluster) is up (after ${i}s)."; break
+ fi
+ if [ "$i" -eq 120 ]; then
+ echo "Server failed to start:"; cat /tmp/connect-server-localcluster.log; exit 1
+ fi
+ sleep 1
+ done
+
+ - name: Parity - resource & artifact tests (local-cluster)
+ id: parity-localcluster
+ env:
+ PYTHONUNBUFFERED: "1"
+ RUST_PYSPARK_SO: ${{ github.workspace }}/python/pyspark/_pyspark.so
+ run: |
+ set -o pipefail
+ # Mirror Apache's local-cluster testnames: resource profiles + artifact
+ # distribution across executors, run through our Rust client. --no-drift-check:
+ # none of these are skiplisted.
+ python scripts/run_official_tests.py \
+ --spark "${{ steps.spark-source.outputs.SPARK_SOURCE }}" \
+ --remote "${SPARK_CONNECT_TESTING_REMOTE}" \
+ --jobs 1 \
+ --no-drift-check \
+ --only "pyspark.resource.tests.test_connect_resources,pyspark.sql.tests.connect.client.test_artifact,pyspark.sql.tests.connect.client.test_artifact_localcluster,pyspark.sql.tests.connect.test_parity_resources" \
+ 2>&1 | tee /tmp/parity_localcluster.log
+
- name: Upload parity log + server log
if: always()
uses: actions/upload-artifact@v4
@@ -212,7 +272,9 @@ jobs:
name: parity-ours-py${{ matrix.python-version }}
path: |
/tmp/parity_ours.log
+ /tmp/parity_localcluster.log
/tmp/connect-server.log
+ /tmp/connect-server-localcluster.log
if-no-files-found: warn
# ---------------------------------------------------------------------------
diff --git a/scripts/run_official_tests.py b/scripts/run_official_tests.py
index f9aee1e..09ad8d4 100755
--- a/scripts/run_official_tests.py
+++ b/scripts/run_official_tests.py
@@ -82,6 +82,28 @@ def discover(spark_py: Path):
return out
+def resolve_only(spec: str, spark_py: Path):
+ """Resolve a comma-separated `--only` list to explicit test-file Paths under spark_py.
+
+ Accepts either a dotted module name the way Apache's `./python/run-tests --testnames`
+ does (`pyspark.resource.tests.test_connect_resources`) or a file path relative to
+ spark_py (`pyspark/sql/tests/connect/client/test_artifact.py`). This is how the
+ local-cluster phase runs a fixed set of resource/artifact tests, including
+ `pyspark.resource.tests.test_connect_resources`, which lives outside the
+ `sql/tests/connect/` scope `discover()` walks.
+ """
+ out = []
+ for name in spec.split(","):
+ name = name.strip()
+ if not name:
+ continue
+ if name.endswith(".py"):
+ out.append(spark_py / name)
+ else:
+ out.append(spark_py.joinpath(*name.split(".")).with_suffix(".py"))
+ return out
+
+
def load_manifest(path: Path):
"""Return (per_file_deselect, whole_file_skip).
@@ -197,13 +219,40 @@ def main():
help="Skip the post-run check that re-runs skiplisted tests to report any that "
"now pass (candidates to remove from the manifest).",
)
+ ap.add_argument(
+ "--only",
+ default="",
+ help="Run ONLY this comma-separated list of tests instead of discovering the "
+ "whole sql/tests/connect suite. Each entry is a dotted module name (like "
+ "Apache's run-tests --testnames, e.g. "
+ "pyspark.resource.tests.test_connect_resources) or a spark_py-relative .py path. "
+ "Used by the local-cluster phase to run the resource/artifact tests, which need "
+ "a real multi-executor cluster.",
+ )
+ ap.add_argument(
+ "--exclude",
+ default="",
+ help="Comma-separated test-file basenames to drop from discovery (e.g. the "
+ "resource/artifact files that only run in the local-cluster phase). Ignored "
+ "when --only is set.",
+ )
args = ap.parse_args()
if not os.environ.get("RUST_PYSPARK_SO"):
print("!! set RUST_PYSPARK_SO to the built extension")
return 2
spark_py = Path(os.path.expanduser(args.spark)) / "python"
- files = discover(spark_py)
+ if args.only:
+ files = resolve_only(args.only, spark_py)
+ missing = [f for f in files if not f.exists()]
+ if missing:
+ print("!! --only test file(s) not found:", *(str(m) for m in missing))
+ return 2
+ else:
+ files = discover(spark_py)
+ if args.exclude:
+ drop = {x.strip() for x in args.exclude.split(",") if x.strip()}
+ files = [f for f in files if f.name not in drop]
if not files:
print("!! no connect test files found under", spark_py)
return 2
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org