You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "ueshin (via GitHub)" <gi...@apache.org> on 2025/11/25 22:51:46 UTC
[PR] [SPARK-54485][SPARK-54494][TESTS] Enable tests in test_parity_udf_profiler and test_parity_memory_profiler [spark]
ueshin opened a new pull request, #53222:
URL: https://github.com/apache/spark/pull/53222
### What changes were proposed in this pull request?
Enables tests in `test_parity_udf_profiler` and `test_parity_memory_profiler`.
Added utility decorators for Spark Connect parity tests.
- `skip_if_server_version_is`
- `skip_if_server_version_is_greater_than_or_equal_to`
### Why are the changes needed?
Some tests are disabled due to Spark Connect cross version tests.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Modified the related tests to use the utility.
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54485][SPARK-54494][TESTS] Re-enable tests in test_parity_udf_profiler and test_parity_memory_profiler [spark]
Posted by "ueshin (via GitHub)" <gi...@apache.org>.
ueshin commented on code in PR #53222:
URL: https://github.com/apache/spark/pull/53222#discussion_r2561839738
##########
python/pyspark/testing/connectutils.py:
##########
@@ -197,3 +198,28 @@ def quiet(self):
return QuietTest(self._legacy_sc)
else:
return contextlib.nullcontext()
+
+
+def skip_if_server_version_is(
Review Comment:
What do you mean by "make it a private one"?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54485][SPARK-54494][TESTS] Re-enable tests in test_parity_udf_profiler and test_parity_memory_profiler [spark]
Posted by "ueshin (via GitHub)" <gi...@apache.org>.
ueshin commented on code in PR #53222:
URL: https://github.com/apache/spark/pull/53222#discussion_r2561839738
##########
python/pyspark/testing/connectutils.py:
##########
@@ -197,3 +198,28 @@ def quiet(self):
return QuietTest(self._legacy_sc)
else:
return contextlib.nullcontext()
+
+
+def skip_if_server_version_is(
Review Comment:
~What do you mean by "make it a private one"?~
I don't intend it to be a private as it's complete usable, but I'm ok to make it a private. cc @HyukjinKwon
##########
python/pyspark/testing/connectutils.py:
##########
@@ -197,3 +198,28 @@ def quiet(self):
return QuietTest(self._legacy_sc)
else:
return contextlib.nullcontext()
+
+
+def skip_if_server_version_is(
Review Comment:
~What do you mean by "make it a private one"?~
I don't intend it to be a private as it's complete usable, but I'm ok to make it a private. WDYT? @HyukjinKwon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54485][SPARK-54494][TESTS] Re-enable tests in test_parity_udf_profiler and test_parity_memory_profiler [spark]
Posted by "ueshin (via GitHub)" <gi...@apache.org>.
ueshin commented on code in PR #53222:
URL: https://github.com/apache/spark/pull/53222#discussion_r2561839738
##########
python/pyspark/testing/connectutils.py:
##########
@@ -197,3 +198,28 @@ def quiet(self):
return QuietTest(self._legacy_sc)
else:
return contextlib.nullcontext()
+
+
+def skip_if_server_version_is(
Review Comment:
~What do you mean by "make it a private one"?~
I don't intend it to be a private as it's complete usable, but I'm ok to make it a private.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54485][SPARK-54494][TESTS] Re-enable tests in test_parity_udf_profiler and test_parity_memory_profiler [spark]
Posted by "ueshin (via GitHub)" <gi...@apache.org>.
ueshin commented on PR #53222:
URL: https://github.com/apache/spark/pull/53222#issuecomment-3578034889
If the decorators look good, I'll submit a PR against master to forward port them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54485][SPARK-54494][TESTS] Re-enable tests in test_parity_udf_profiler and test_parity_memory_profiler [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on code in PR #53222:
URL: https://github.com/apache/spark/pull/53222#discussion_r2561833868
##########
python/pyspark/testing/connectutils.py:
##########
@@ -197,3 +198,28 @@ def quiet(self):
return QuietTest(self._legacy_sc)
else:
return contextlib.nullcontext()
+
+
+def skip_if_server_version_is(
Review Comment:
this seems to be a helper decorator. should we make it a private one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54485][SPARK-54494][PYTHON][TESTS] Re-enable tests in test_parity_udf_profiler and test_parity_memory_profiler [spark]
Posted by "ueshin (via GitHub)" <gi...@apache.org>.
ueshin commented on PR #53222:
URL: https://github.com/apache/spark/pull/53222#issuecomment-3578059322
#53223 for forward-port.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54485][SPARK-54494][TESTS] Re-enable tests in test_parity_udf_profiler and test_parity_memory_profiler [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #53222:
URL: https://github.com/apache/spark/pull/53222#discussion_r2561929359
##########
python/pyspark/testing/connectutils.py:
##########
@@ -197,3 +198,28 @@ def quiet(self):
return QuietTest(self._legacy_sc)
else:
return contextlib.nullcontext()
+
+
+def skip_if_server_version_is(
Review Comment:
It's fine. let's just leave it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54485][SPARK-54494][PYTHON][TESTS][4.0] Re-enable tests in test_parity_udf_profiler and test_parity_memory_profiler [spark]
Posted by "ueshin (via GitHub)" <gi...@apache.org>.
ueshin commented on PR #53222:
URL: https://github.com/apache/spark/pull/53222#issuecomment-3583474124
The failed tests are `Run / Build modules: pyspark-mllib, pyspark-ml, pyspark-ml-connect`.
They are not related to this PR, and can't pass on `branch-4.0` in my GitHub Action for some reason same as before.
Retrying just in case:
- https://github.com/ueshin/apache-spark/actions/runs/19714734699/job/56498634043
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54485][SPARK-54494][PYTHON][TESTS][4.0] Re-enable tests in test_parity_udf_profiler and test_parity_memory_profiler [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #53222: [SPARK-54485][SPARK-54494][PYTHON][TESTS][4.0] Re-enable tests in test_parity_udf_profiler and test_parity_memory_profiler
URL: https://github.com/apache/spark/pull/53222
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org