You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "sryza (via GitHub)" <gi...@apache.org> on 2025/05/19 23:45:14 UTC
[PR] [SPARK-52224] Introduce pyyaml as a dependency for the Python client [spark]
sryza opened a new pull request, #50944:
URL: https://github.com/apache/spark/pull/50944
### What changes were proposed in this pull request?
Introduces pyyaml as a dependency for the Python client. When `pip install`-ing the pyspark client, it will be installed with it.
### Why are the changes needed?
The pipeline spec file described in the [Declarative Pipelines SPIP](https://docs.google.com/document/d/1PsSTngFuRVEOvUGzp_25CQL1yfzFHFr02XdMfQ7jOM4/edit?tab=t.0) expects data in a YAML format. YAML is superior to alternatives, for a few reasons:
- Unlike the flat files that are used for [spark-submit confs](https://spark.apache.org/docs/latest/submitting-applications.html#loading-configuration-from-a-file), it supports the hierarchical data required by the pipeline spec.
- It's much more user-friendly to author than JSON.
- It's consistent with the config files used for similar tools, like dbt.
The Declarative Pipelines CLI will be a Spark Connect Python client, and thus require a Python library for loading YAML. The pyyaml library is an extremely stable dependency. The `safe_load` function that we'll use to load YAML files was introduced more than a decade ago.
### Does this PR introduce _any_ user-facing change?
Yes – users who `pip install` the PySpark client library will see the pyyaml library installed.
### How was this patch tested?
- Made a clean virtualenv
- Ran `pip install python/packaging/client`
- Confirmed that I could `import yaml` in a Python shell
### Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52224][CONNECT][PYTHON] Introduce pyyaml as a dependency for the Python client [spark]
Posted by "xinrong-meng (via GitHub)" <gi...@apache.org>.
xinrong-meng commented on PR #50944:
URL: https://github.com/apache/spark/pull/50944#issuecomment-2902061024
LGTM thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52224][CONNECT][PYTHON] Introduce pyyaml as a dependency for the Python client [spark]
Posted by "sryza (via GitHub)" <gi...@apache.org>.
sryza commented on PR #50944:
URL: https://github.com/apache/spark/pull/50944#issuecomment-2894702907
Thanks @grundprinzip – I added those too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52224][CONNECT][PYTHON] Introduce pyyaml as a dependency for the Python client [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #50944:
URL: https://github.com/apache/spark/pull/50944#issuecomment-2899816497
@sryza please re-run the failed tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52224][CONNECT][PYTHON] Introduce pyyaml as a dependency for the Python client [spark]
Posted by "sryza (via GitHub)" <gi...@apache.org>.
sryza closed pull request #50944: [SPARK-52224][CONNECT][PYTHON] Introduce pyyaml as a dependency for the Python client
URL: https://github.com/apache/spark/pull/50944
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52224][CONNECT][PYTHON] Introduce pyyaml as a dependency for the Python client [spark]
Posted by "sryza (via GitHub)" <gi...@apache.org>.
sryza commented on PR #50944:
URL: https://github.com/apache/spark/pull/50944#issuecomment-2905734433
Thanks for catching this @dongjoon-hyun. I'll create a followup for that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52224][CONNECT][PYTHON] Introduce pyyaml as a dependency for the Python client [spark]
Posted by "sryza (via GitHub)" <gi...@apache.org>.
sryza commented on PR #50944:
URL: https://github.com/apache/spark/pull/50944#issuecomment-2956962734
Here's the followup: https://github.com/apache/spark/pull/51138
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org