You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "zhengruifeng (via GitHub)" <gi...@apache.org> on 2025/02/10 04:06:20 UTC
[PR] [SPARK-51143][PYTHON] Pin `plotly==5.24.1` [spark]
zhengruifeng opened a new pull request, #49863:
URL: https://github.com/apache/spark/pull/49863
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error type or message, please read the guideline first in
'common/utils/src/main/resources/error/README.md'.
-->
### What changes were proposed in this pull request?
Pin `plotly==5.24.1`
### Why are the changes needed?
the latest plotlly 6.0 has causes many plot-related test failures
### Does this PR introduce _any_ user-facing change?
no
### How was this patch tested?
manually checked with
```
python/run-tests -k --python-executables python3 --testnames 'pyspark.sql.tests.connect.test_parity_frame_plot_plotly FramePlotPlotlyParityTests.test_pie_plot'
```
before:
```
(spark_312) ➜ spark git:(pin_plotly) python/run-tests -k --python-executables python3 --testnames 'pyspark.sql.tests.connect.test_parity_frame_plot_plotly FramePlotPlotlyParityTests.test_pie_plot'
Running PySpark tests. Output is in /Users/ruifeng.zheng/Dev/spark/python/unit-tests.log
Will test against the following Python executables: ['python3']
Will test the following Python tests: ['pyspark.sql.tests.connect.test_parity_frame_plot_plotly FramePlotPlotlyParityTests.test_pie_plot']
python3 python_implementation is CPython
python3 version is: Python 3.12.9
Starting test(python3): pyspark.sql.tests.connect.test_parity_frame_plot_plotly FramePlotPlotlyParityTests.test_pie_plot (temp output: /Users/ruifeng.zheng/Dev/spark/python/target/4d10075d-bb7b-4d4b-b17d-edbef2f22227/python3__pyspark.sql.tests.connect.test_parity_frame_plot_plotly_FramePlotPlotlyParityTests.test_pie_plot__6qxzu16x.log)
Running tests...
----------------------------------------------------------------------
WARNING: Using incubator modules: jdk.incubator.vector
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
/Users/ruifeng.zheng/Dev/spark/python/pyspark/sql/connect/conf.py:64: UserWarning: Failed to set spark.connect.execute.reattachable.senderMaxStreamDuration to Some(1s) due to [CANNOT_MODIFY_CONFIG] Cannot modify the value of the Spark config: "spark.connect.execute.reattachable.senderMaxStreamDuration".
See also 'https://spark.apache.org/docs/latest/sql-migration-guide.html#ddl-statements'. SQLSTATE: 46110
warnings.warn(warn)
/Users/ruifeng.zheng/Dev/spark/python/pyspark/sql/connect/conf.py:64: UserWarning: Failed to set spark.connect.execute.reattachable.senderMaxStreamSize to Some(123) due to [CANNOT_MODIFY_CONFIG] Cannot modify the value of the Spark config: "spark.connect.execute.reattachable.senderMaxStreamSize".
See also 'https://spark.apache.org/docs/latest/sql-migration-guide.html#ddl-statements'. SQLSTATE: 46110
warnings.warn(warn)
test_pie_plot (pyspark.sql.tests.connect.test_parity_frame_plot_plotly.FramePlotPlotlyParityTests.test_pie_plot) ... FAIL (1.760s)
======================================================================
FAIL [1.760s]: test_pie_plot (pyspark.sql.tests.connect.test_parity_frame_plot_plotly.FramePlotPlotlyParityTests.test_pie_plot)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/ruifeng.zheng/Dev/spark/python/pyspark/sql/tests/plot/test_frame_plot_plotly.py", line 318, in test_pie_plot
self._check_fig_data(fig["data"][0], **expected_fig_data_sales)
File "/Users/ruifeng.zheng/Dev/spark/python/pyspark/sql/tests/plot/test_frame_plot_plotly.py", line 81, in _check_fig_data
self.assertEqual(converted_values, expected_value)
AssertionError: Lists differ: [1517356800000000000, 1519776000000000000,[37 chars]0000] != [datetime.datetime(2018, 1, 31, 0, 0), dat[105 chars], 0)]
First differing element 0:
datetime.datetime(2018, 1, 31, 0, 0)
- [1517356800000000000,
- 1519776000000000000,
- 1522454400000000000,
- 1525046400000000000]
+ [datetime.datetime(2018, 1, 31, 0, 0),
+ datetime.datetime(2018, 2, 28, 0, 0),
+ datetime.datetime(2018, 3, 31, 0, 0),
+ datetime.datetime(2018, 4, 30, 0, 0)]
----------------------------------------------------------------------
Ran 1 test in 5.573s
FAILED (failures=1)
Generating XML reports...
Generated XML report: target/test-reports/TEST-pyspark.sql.tests.connect.test_parity_frame_plot_plotly.FramePlotPlotlyParityTests-20250210120410.xml
Had test failures in pyspark.sql.tests.connect.test_parity_frame_plot_plotly FramePlotPlotlyParityTests.test_pie_plot with python3; see logs.
```
after:
```
(spark_312) ➜ spark git:(pin_plotly) python/run-tests -k --python-executables python3 --testnames 'pyspark.sql.tests.connect.test_parity_frame_plot_plotly FramePlotPlotlyParityTests.test_pie_plot'
Running PySpark tests. Output is in /Users/ruifeng.zheng/Dev/spark/python/unit-tests.log
Will test against the following Python executables: ['python3']
Will test the following Python tests: ['pyspark.sql.tests.connect.test_parity_frame_plot_plotly FramePlotPlotlyParityTests.test_pie_plot']
python3 python_implementation is CPython
python3 version is: Python 3.12.9
Starting test(python3): pyspark.sql.tests.connect.test_parity_frame_plot_plotly FramePlotPlotlyParityTests.test_pie_plot (temp output: /Users/ruifeng.zheng/Dev/spark/python/target/28a7ea8a-6249-4b16-9831-d785b1af2254/python3__pyspark.sql.tests.connect.test_parity_frame_plot_plotly_FramePlotPlotlyParityTests.test_pie_plot__eezgr0hf.log)
Finished test(python3): pyspark.sql.tests.connect.test_parity_frame_plot_plotly FramePlotPlotlyParityTests.test_pie_plot (5s)
Tests passed in 5 seconds
```
### Was this patch authored or co-authored using generative AI tooling?
no
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51143][PYTHON] Pin `plotly<6.0.0` and `torch<2.6.0` [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #49863:
URL: https://github.com/apache/spark/pull/49863#issuecomment-2649549622
Merged to master and branch-4.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51143][PYTHON] Pin `plotly<6.0.0` and `torch<2.6.0` [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #49863: [SPARK-51143][PYTHON] Pin `plotly<6.0.0` and `torch<2.6.0`
URL: https://github.com/apache/spark/pull/49863
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51143][PYTHON] Pin `plotly==5.24.1` [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #49863:
URL: https://github.com/apache/spark/pull/49863#issuecomment-2646873479
also cc @cloud-fan we probably need to pin `plotly` in the release docker image
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51143][PYTHON] Pin `plotly==5.24.1` [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #49863:
URL: https://github.com/apache/spark/pull/49863#issuecomment-2647094992
update in the docker file trigger the refresh of the cache, and `torch` is also upgraded and caused
```
======================================================================
ERROR [42.116s]: test_save_load (pyspark.ml.tests.connect.test_connect_classification.ClassificationTestsOnConnect.test_save_load)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/__w/spark/spark/python/pyspark/ml/tests/connect/test_legacy_mode_classification.py", line 185, in test_save_load
lor_torch_model = torch.load(
^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/torch/serialization.py", line 1470, in load
raise pickle.UnpicklingError(_get_wo_message(str(e))) from None
_pickle.UnpicklingError: Weights only load failed. This file can still be loaded, to do so you have two options, do those steps only if you trust the source of the checkpoint.
(1) In PyTorch 2.6, we changed the default value of the `weights_only` argument in `torch.load` from `False` to `True`. Re-running `torch.load` with `weights_only` set to `False` will likely succeed, but it can result in arbitrary code execution. Do it only if you got the file from a trusted source.
(2) Alternatively, to load with `weights_only=True` please check the recommended steps in the following error message.
WeightsUnpickler error: Unsupported global: GLOBAL torch.nn.modules.container.Sequential was not an allowed global by default. Please use `torch.serialization.add_safe_globals([Sequential])` or the `torch.serialization.safe_globals([Sequential])` context manager to allowlist this global if you trust this class/function.
Check the documentation of torch.load to learn more about types accepted by default with weights_only https://pytorch.org/docs/stable/generated/torch.load.html.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51143][PYTHON] Pin `plotly<6.0.0` and `torch<2.6.0` [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #49863:
URL: https://github.com/apache/spark/pull/49863#issuecomment-2647317532
set to draft first to avoid accidental merging due to the existence of build failures
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org