You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "itholic (via GitHub)" <gi...@apache.org> on 2023/09/21 05:12:07 UTC
[GitHub] [spark] itholic opened a new pull request, #43024: [SPARK-45246][BUILD][PS] Encourage using latest `jinja2` other than documentation build
itholic opened a new pull request, #43024:
URL: https://github.com/apache/spark/pull/43024
### What changes were proposed in this pull request?
This PR proposes to update the `jinja2` version from `dev/requirements.txt` to the latest version for PySpark.
### Why are the changes needed?
Currently, `jinja2<3.0.0` is installed when running `pip install -r dev/requirements.txt`, but it's not working for some functionality for Pandas API on Spark.
Because Pandas requires the latest version of `jinja2` for their LaTex render engine, so Pandas API on Spark also requires the same version since we're leverage the Pandas internally.
<img width="728" alt="Screenshot 2023-09-21 at 2 06 23 PM" src="https://github.com/apache/spark/assets/44108233/1f0bbb15-829c-473b-bb47-e602fa6cb04d">
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
No.
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
If benchmark tests were added, please run the benchmarks in GitHub Actions for the consistent environment, and the instructions could accord to: https://spark.apache.org/developer-tools.html#github-workflow-benchmarks.
-->
The existing CI should pass.
### Was this patch authored or co-authored using generative AI tooling?
<!--
If generative AI tooling has been used in the process of authoring this patch, please include the
phrase: 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on a diff in pull request #43024: [SPARK-45246][BUILD][PS] Encourage using latest `jinja2` other than documentation build
Posted by "itholic (via GitHub)" <gi...@apache.org>.
itholic commented on code in PR #43024:
URL: https://github.com/apache/spark/pull/43024#discussion_r1333842788
##########
dev/requirements.txt:
##########
@@ -34,7 +35,9 @@ pydata_sphinx_theme
ipython
nbsphinx
numpydoc
-jinja2<3.0.0
+# Commented this because we need the latest version of jinja2 for Pandas API on Spark.
+# Please manually install jinja2<3.0.0 when building the documentation.
+# jinja2<3.0.0
Review Comment:
Yeah, I tried to make the build work with jinja2 3.0.0+ at the first time, but there is compatibility issues with Sphinx version we're currently using.
Let me just close this and revisit when the Sphinx support better compatibility with the jinja2 3.0.0+.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #43024: [SPARK-45246][BUILD][PS] Encourage using latest `jinja2` other than documentation build
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #43024:
URL: https://github.com/apache/spark/pull/43024#discussion_r1333832303
##########
dev/requirements.txt:
##########
@@ -34,7 +35,9 @@ pydata_sphinx_theme
ipython
nbsphinx
numpydoc
-jinja2<3.0.0
+# Commented this because we need the latest version of jinja2 for Pandas API on Spark.
+# Please manually install jinja2<3.0.0 when building the documentation.
+# jinja2<3.0.0
Review Comment:
I think we should probably just leave it as is for now, or let the documentation build works with jinja2 3.0.0+
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #43024: [SPARK-45246][BUILD][PS] Encourage using latest `jinja2` other than documentation build
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #43024:
URL: https://github.com/apache/spark/pull/43024#discussion_r1332556446
##########
dev/requirements.txt:
##########
@@ -34,7 +35,9 @@ pydata_sphinx_theme
ipython
nbsphinx
numpydoc
-jinja2<3.0.0
+# Commented this because we need the latest version of jinja2 for Pandas API on Spark.
+# Please manually install jinja2<3.0.0 when building the documentation.
+# jinja2<3.0.0
Review Comment:
> but it's not working for some functionality for Pandas API on Spark from 4.0.0 release.
Which functionality does it not work? I think we should better have one version for development if possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic closed pull request #43024: [SPARK-45246][BUILD][PS] Encourage using latest `jinja2` other than documentation build
Posted by "itholic (via GitHub)" <gi...@apache.org>.
itholic closed pull request #43024: [SPARK-45246][BUILD][PS] Encourage using latest `jinja2` other than documentation build
URL: https://github.com/apache/spark/pull/43024
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on a diff in pull request #43024: [SPARK-45246][BUILD][PS] Encourage using latest `jinja2` other than documentation build
Posted by "itholic (via GitHub)" <gi...@apache.org>.
itholic commented on code in PR #43024:
URL: https://github.com/apache/spark/pull/43024#discussion_r1332728611
##########
dev/requirements.txt:
##########
@@ -34,7 +35,9 @@ pydata_sphinx_theme
ipython
nbsphinx
numpydoc
-jinja2<3.0.0
+# Commented this because we need the latest version of jinja2 for Pandas API on Spark.
+# Please manually install jinja2<3.0.0 when building the documentation.
+# jinja2<3.0.0
Review Comment:
`(DataFrame|Series).to_latex` does not work without `jinja2>=3.1.2`
```python
>>> import pyspark.pandas as ps
>>> psdf = ps.DataFrame({"A": [1, 2, 3]})
>>> psdf.to_latex()
Traceback (most recent call last):
...
ImportError: Pandas requires version '3.1.2' or newer of 'jinja2' (version '2.11.3' currently installed).
```
But I'm okay we just keep the current status, because this is the only case for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org