You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "zhengruifeng (via GitHub)" <gi...@apache.org> on 2026/04/09 04:32:17 UTC
[PR] [SPARK-XXXXX][PYTHON][TESTS] Skip Kafka streaming RTM tests when dependencies are not installed [spark]
zhengruifeng opened a new pull request, #55270:
URL: https://github.com/apache/spark/pull/55270
### What changes were proposed in this pull request?
Skip `StreamingKafkaTests` in `test_streaming_kafka_rtm.py` when `testcontainers` or `kafka-python` is not installed, instead of raising an `ImportError` that crashes the entire test module.
### Why are the changes needed?
The CI job `pyspark-structured-streaming` fails because `test_streaming_kafka_rtm.py` raises `ImportError` at module level when `testcontainers` is not installed. This causes the entire test runner to exit with a non-zero code, failing the CI build. See: https://github.com/apache/spark/actions/runs/24128422095/job/70399041605
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
The test class is now decorated with `@unittest.skipIf(not have_kafka_deps, ...)`, which gracefully skips when the dependencies are missing — the same pattern used for Docker availability in the same file.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55306][PYTHON][TESTS][FOLLOW-UP] Skip Kafka streaming RTM tests when dependencies are not installed [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #55270:
URL: https://github.com/apache/spark/pull/55270#issuecomment-4211771772
@jerrypeng because you didn't install the dependency in all scheduled workflow, e.g.
https://github.com/apache/spark/actions/workflows/build_python_3.12
https://github.com/apache/spark/actions/workflows/build_python_minimum.yml
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55306][PYTHON][TESTS][FOLLOW-UP] Skip Kafka streaming RTM tests when dependencies are not installed [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #55270:
URL: https://github.com/apache/spark/pull/55270#issuecomment-4211631934
> This causes the entire test runner to exit with a non-zero code, failing the CI build. See: https://github.com/apache/spark/actions/runs/24128422095/job/70399041605
The PR added the test code passed all CIs. Is the failed pipeline not running on the PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55306][PYTHON][TESTS][FOLLOW-UP] Skip Kafka streaming RTM tests when dependencies are not installed [spark]
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on PR #55270:
URL: https://github.com/apache/spark/pull/55270#issuecomment-4211571066
https://github.com/apache/spark/pull/55266
Can we check whether this PR is doing the same work?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55306][PYTHON][TESTS][FOLLOW-UP] Skip Kafka streaming RTM tests when dependencies are not installed [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng closed pull request #55270: [SPARK-55306][PYTHON][TESTS][FOLLOW-UP] Skip Kafka streaming RTM tests when dependencies are not installed
URL: https://github.com/apache/spark/pull/55270
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55306][PYTHON][TESTS][FOLLOW-UP] Skip Kafka streaming RTM tests when dependencies are not installed [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #55270:
URL: https://github.com/apache/spark/pull/55270#issuecomment-4211662208
@HeartSaVioR just notice https://github.com/apache/spark/pull/55266, I think we don't need this PR now.
@viirya
> Is the failed pipeline not running on the PR?
yes, it is failed on scheduled workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55306][PYTHON][TESTS][FOLLOW-UP] Skip Kafka streaming RTM tests when dependencies are not installed [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #55270:
URL: https://github.com/apache/spark/pull/55270#issuecomment-4211553330
cc @jerrypeng
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55306][PYTHON][TESTS][FOLLOW-UP] Skip Kafka streaming RTM tests when dependencies are not installed [spark]
Posted by "jerrypeng (via GitHub)" <gi...@apache.org>.
jerrypeng commented on PR #55270:
URL: https://github.com/apache/spark/pull/55270#issuecomment-4211753854
> yes, it is failed on scheduled workflows.
@zhengruifeng why would it succeed for PRs but fail for scheduled workflows?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org