You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/11/26 03:18:11 UTC
[GitHub] [spark] Yikun opened a new pull request #34717: Bump minimum pandas version to 1.0.0
Yikun opened a new pull request #34717:
URL: https://github.com/apache/spark/pull/34717
### What changes were proposed in this pull request?
Bump minimum pandas version to 1.0.0
### Why are the changes needed?
To be done
### Does this PR introduce _any_ user-facing change?
To be done
### How was this patch tested?
To be done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun edited a comment on pull request #34717: Bump minimum pandas version to 1.0.0
Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-979659317
Just to start the discussion, by using below sql according [1], we can got the all download stat of Pandas in last 3 months.
```SQL
SELECT
file.version AS file_version,
COUNT(*) AS num_downloads,
FROM `the-psf.pypi.file_downloads`
WHERE file.project = 'pandas'
AND
-- Only query the last 3 months of history
DATE(timestamp)
BETWEEN DATE_SUB(CURRENT_DATE(), INTERVAL 3 MONTH)
AND CURRENT_DATE()
GROUP BY `file_version`
ORDER BY `num_downloads` DESC
```
Here is the Top 20 data, about 77% of the overall data:
| version | number | percent
-- | -- | -- | --
1 | 0.25.3 | 35149221 | 14.28%
2 | 1.1.5 | 28722806 | 11.67%
3 | 1.3.4 | 20944236 | 8.51%
4 | 1.3.3 | 16861573 | 6.85%
5 | 0.24.2 | 13235233 | 5.38%
6 | 1.0.5 | 9201989 | 3.74%
7 | 1.3.2 | 9077326 | 3.69%
8 | 1.2.5 | 7902532 | 3.21%
9 | 1.2.4 | 5754284 | 2.34%
10 | 1.1.4 | 5710439 | 2.32%
11 | 1.1.0 | 4760847 | 1.93%
12 | 1.1.2 | 4621441 | 1.88%
13 | 1.2.3 | 4607043 | 1.87%
14 | 1.0.3 | 4601230 | 1.87%
15 | 0.23.4 | 4251044 | 1.73%
16 | 0.25.0 | 3862673 | 1.57%
17 | 1.2.1 | 2952346 | 1.20%
18 | 1.0.1 | 2690006 | 1.09%
19 | 0.22.0 | 2680710 | 1.09%
20 | 1.2.0 | 2645339 | 1.07%
21 | 0.24.1 | 2635411 | 1.07%
[1] https://packaging.python.org/guides/analyzing-pypi-package-downloads/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun edited a comment on pull request #34717: Bump minimum pandas version to 1.0.0
Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-979659317
Just to start the discussion, by using below sql according [1], we can got the all download stat of Pandas in last 3 months.
```SQL
SELECT
file.version AS file_version,
COUNT(*) AS num_downloads,
FROM `the-psf.pypi.file_downloads`
WHERE file.project = 'pandas'
AND
-- Only query the last 3 months of history
DATE(timestamp)
BETWEEN DATE_SUB(CURRENT_DATE(), INTERVAL 3 MONTH)
AND CURRENT_DATE()
GROUP BY `file_version`
ORDER BY `num_downloads` DESC
```
Here is the Top 20 data, about 77% of the overall data, complete result can be found in [here](https://gist.github.com/Yikun/a1d8168c316966db520cf8f8a43ff0bf):
| version | number | percent
-- | -- | -- | --
1 | 0.25.3 | 35149221 | 14.28%
2 | 1.1.5 | 28722806 | 11.67%
3 | 1.3.4 | 20944236 | 8.51%
4 | 1.3.3 | 16861573 | 6.85%
5 | 0.24.2 | 13235233 | 5.38%
6 | 1.0.5 | 9201989 | 3.74%
7 | 1.3.2 | 9077326 | 3.69%
8 | 1.2.5 | 7902532 | 3.21%
9 | 1.2.4 | 5754284 | 2.34%
10 | 1.1.4 | 5710439 | 2.32%
11 | 1.1.0 | 4760847 | 1.93%
12 | 1.1.2 | 4621441 | 1.88%
13 | 1.2.3 | 4607043 | 1.87%
14 | 1.0.3 | 4601230 | 1.87%
15 | 0.23.4 | 4251044 | 1.73%
16 | 0.25.0 | 3862673 | 1.57%
17 | 1.2.1 | 2952346 | 1.20%
18 | 1.0.1 | 2690006 | 1.09%
19 | 0.22.0 | 2680710 | 1.09%
20 | 1.2.0 | 2645339 | 1.07%
21 | 0.24.1 | 2635411 | 1.07%
[1] https://packaging.python.org/guides/analyzing-pypi-package-downloads/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5
Posted by GitBox <gi...@apache.org>.
Yikun commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-981564669
As a conclusion in here:
- before v1.0.1: test failed due to https://github.com/apache/spark/pull/34717#issuecomment-981182023 https://github.com/apache/spark/pull/34717#issuecomment-981182023
- before v1.0.5: test failed due to https://github.com/apache/spark/pull/34717#issuecomment-981463864
- There are only a precision problem on v1.0.5, so just add `almost=True` to `assert_eq`, we can fix the testcase
So, I bump minimum pandas version to v1.0.5, the v1.0.5 is also the latest version of [Pandas verions 1.0](https://pandas.pydata.org/docs/whatsnew/index.html#version-1-0).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-982288646
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50215/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34717: Bump minimum pandas version to 1.0.0
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-979683584
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50117/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun edited a comment on pull request #34717: Bump minimum pandas version to 1.0.0
Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-979659317
Just to start the discussion, by using below sql according [1], we can got the all download stat of Pandas in last 3 months.
```SQL
SELECT
file.version AS file_version,
COUNT(*) AS num_downloads,
FROM `the-psf.pypi.file_downloads`
WHERE file.project = 'pandas'
AND
-- Only query the last 3 months of history
DATE(timestamp)
BETWEEN DATE_SUB(CURRENT_DATE(), INTERVAL 3 MONTH)
AND CURRENT_DATE()
GROUP BY `file_version`
ORDER BY `num_downloads` DESC
```
Here is the Top 20 data, about 77% of the overall data, complete result can be found in [here](https://gist.github.com/Yikun/a1d8168c316966db520cf8f8a43ff0bf):
| version | number | percent
-- | -- | -- | --
1 | 0.25.3 | 35149221 | 14.28%
2 | 1.1.5 | 28722806 | 11.67%
3 | 1.3.4 | 20944236 | 8.51%
4 | 1.3.3 | 16861573 | 6.85%
5 | 0.24.2 | 13235233 | 5.38%
6 | 1.0.5 | 9201989 | 3.74%
7 | 1.3.2 | 9077326 | 3.69%
8 | 1.2.5 | 7902532 | 3.21%
9 | 1.2.4 | 5754284 | 2.34%
10 | 1.1.4 | 5710439 | 2.32%
11 | 1.1.0 | 4760847 | 1.93%
12 | 1.1.2 | 4621441 | 1.88%
13 | 1.2.3 | 4607043 | 1.87%
14 | 1.0.3 | 4601230 | 1.87%
15 | 0.23.4 | 4251044 | 1.73%
16 | 0.25.0 | 3862673 | 1.57%
17 | 1.2.1 | 2952346 | 1.20%
18 | 1.0.1 | 2690006 | 1.09%
19 | 0.22.0 | 2680710 | 1.09%
20 | 1.2.0 | 2645339 | 1.07%
21 | 0.24.1 | 2635411 | 1.07%
- There are more than 60+% users downloaded the 1.x version in last 3 months
- There are 26+% users downloaded the 0.23.2~1.0
[1] https://packaging.python.org/guides/analyzing-pypi-package-downloads/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #34717:
URL: https://github.com/apache/spark/pull/34717#discussion_r758572738
##########
File path: python/docs/source/user_guide/sql/arrow_pandas.rst
##########
@@ -387,7 +387,7 @@ working with timestamps in ``pandas_udf``\s to get the best performance, see
Recommended Pandas and PyArrow Versions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-For usage with pyspark.sql, the minimum supported versions of Pandas is 0.23.2 and PyArrow is 1.0.0.
+For usage with pyspark.sql, the minimum supported versions of Pandas is 1.0.5 and PyArrow is 1.0.0.
Review comment:
Should we mention there are some issues with versions like 1.0.0, 1.0.1?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on a change in pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5
Posted by GitBox <gi...@apache.org>.
Yikun commented on a change in pull request #34717:
URL: https://github.com/apache/spark/pull/34717#discussion_r758902796
##########
File path: python/pyspark/pandas/tests/test_series.py
##########
@@ -2209,12 +2209,12 @@ def test_mad(self):
pser = pd.Series([1, 2, 3, 4], name="Koalas")
psser = ps.from_pandas(pser)
- self.assert_eq(pser.mad(), psser.mad())
+ self.assert_eq(pser.mad(), psser.mad(), almost=True)
Review comment:
Addressed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.0
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-980488878
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50141/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on a change in pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5
Posted by GitBox <gi...@apache.org>.
Yikun commented on a change in pull request #34717:
URL: https://github.com/apache/spark/pull/34717#discussion_r758898358
##########
File path: python/docs/source/user_guide/sql/arrow_pandas.rst
##########
@@ -387,7 +387,7 @@ working with timestamps in ``pandas_udf``\s to get the best performance, see
Recommended Pandas and PyArrow Versions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-For usage with pyspark.sql, the minimum supported versions of Pandas is 0.23.2 and PyArrow is 1.0.0.
+For usage with pyspark.sql, the minimum supported versions of Pandas is 1.0.5 and PyArrow is 1.0.0.
Review comment:
How about?
For usage with pyspark.sql, the minimum supported versions of Pandas is 1.0.5 and PyArrow is 1.0.0. Lower versions (such as there are some known issues under with v1.0.0, v1.0.1, see more in [link](https://github.com/apache/spark/pull/34717)) or higher versions may be used, however, compatibility and data correctness can not be guaranteed and should be verified by the user.
Maybe need more suggestion from native speaker. T_T, and if it's necessary we could do it in next commits in this PR or followup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on a change in pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5
Posted by GitBox <gi...@apache.org>.
Yikun commented on a change in pull request #34717:
URL: https://github.com/apache/spark/pull/34717#discussion_r758894595
##########
File path: python/pyspark/pandas/tests/test_series.py
##########
@@ -2209,12 +2209,12 @@ def test_mad(self):
pser = pd.Series([1, 2, 3, 4], name="Koalas")
psser = ps.from_pandas(pser)
- self.assert_eq(pser.mad(), psser.mad())
+ self.assert_eq(pser.mad(), psser.mad(), almost=True)
Review comment:
Yep, I tried `self.assert_eq(pser.mad(), psser.mad(), check_exact=False)`, it failed again. Because the test.mad() result value type is a numpy.float64 scalar, it would be matched to:
https://github.com/apache/spark/blob/c38c617bcd04429e50310e4c7e0bd989c6555836/python/pyspark/testing/pandasutils.py#L237-L240
So, only `almost=True` work.
And I know your concern to make sure the type is right in here, but not only to check value. I notice that only last check is failed in my env, so we can keep top 3 check as original, and just make last check as almost=True, and also add a note in here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.0
Posted by GitBox <gi...@apache.org>.
Yikun commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-981463864
Complete all `pyspark-pandas-slow` test with:
```
python/run-tests --modules=pyspark-pandas --parallelism=2 --python-executable=python3
```
Serveral test cases failed in 1.0.1 due to `AttributeError: type object 'object' has no attribute 'dtype'` and **passed with pandas v1.0.5**.
<details>
<summary>Test failure details</summary>
```
======================================================================
ERROR: test_astype (pyspark.pandas.tests.data_type_ops.test_categorical_ops.CategoricalOpsTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/tests/data_type_ops/test_categorical_ops.py", line 204, in test_astype
self.assert_eq(pser.astype(int), psser.astype(int))
File "/Users/jiangyikun/spark/spark/python/pyspark/testing/pandasutils.py", line 224, in assert_eq
robj = self._to_pandas(right)
File "/Users/jiangyikun/spark/spark/python/pyspark/testing/pandasutils.py", line 245, in _to_pandas
return obj.to_pandas()
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/series.py", line 1588, in to_pandas
return self._to_pandas()
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/series.py", line 1594, in _to_pandas
return self._to_internal_pandas().copy()
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/series.py", line 6349, in _to_internal_pandas
return self._psdf._internal.to_pandas_frame[self.name]
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/utils.py", line 584, in wrapped_lazy_property
setattr(self, attr_name, fn(self))
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/internal.py", line 1049, in to_pandas_frame
pdf = sdf.toPandas()
File "/Users/jiangyikun/spark/spark/python/pyspark/sql/pandas/conversion.py", line 185, in toPandas
pdf = pd.DataFrame(columns=tmp_column_names).astype(
File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/frame.py", line 435, in __init__
mgr = init_dict(data, index, columns, dtype=dtype)
File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/internals/construction.py", line 239, in init_dict
val = construct_1d_arraylike_from_scalar(np.nan, len(index), nan_dtype)
File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/dtypes/cast.py", line 1449, in construct_1d_arraylike_from_scalar
dtype = dtype.dtype
AttributeError: type object 'object' has no attribute 'dtype'
======================================================================
ERROR: test_read_csv (pyspark.pandas.tests.test_csv.CsvTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/tests/test_csv.py", line 151, in test_read_csv
check(usecols=[])
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/tests/test_csv.py", line 138, in check
self.assert_eq(expected, actual, almost=True)
File "/Users/jiangyikun/spark/spark/python/pyspark/testing/pandasutils.py", line 224, in assert_eq
robj = self._to_pandas(right)
File "/Users/jiangyikun/spark/spark/python/pyspark/testing/pandasutils.py", line 245, in _to_pandas
return obj.to_pandas()
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/frame.py", line 4856, in to_pandas
return self._to_pandas()
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/frame.py", line 4862, in _to_pandas
return self._internal.to_pandas_frame.copy()
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/utils.py", line 584, in wrapped_lazy_property
setattr(self, attr_name, fn(self))
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/internal.py", line 1049, in to_pandas_frame
pdf = sdf.toPandas()
File "/Users/jiangyikun/spark/spark/python/pyspark/sql/pandas/conversion.py", line 185, in toPandas
pdf = pd.DataFrame(columns=tmp_column_names).astype(
File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/frame.py", line 435, in __init__
mgr = init_dict(data, index, columns, dtype=dtype)
File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/internals/construction.py", line 239, in init_dict
val = construct_1d_arraylike_from_scalar(np.nan, len(index), nan_dtype)
File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/dtypes/cast.py", line 1449, in construct_1d_arraylike_from_scalar
dtype = dtype.dtype
AttributeError: type object 'object' has no attribute 'dtype'
======================================================================
ERROR: test_kde_plot (pyspark.pandas.tests.plot.test_frame_plot_plotly.DataFramePlotPlotlyTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/tests/plot/test_frame_plot_plotly.py", line 262, in test_kde_plot
actual = psdf.plot.kde(bw_method=5, ind=3)
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/plot/core.py", line 946, in kde
return self(kind="kde", bw_method=bw_method, ind=ind, **kwargs)
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/plot/core.py", line 498, in __call__
return plot_backend.plot_pandas_on_spark(plot_data, kind=kind, **kwargs)
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/plot/plotly.py", line 44, in plot_pandas_on_spark
return plot_kde(data, **kwargs)
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/plot/plotly.py", line 202, in plot_kde
pd.DataFrame(
File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/frame.py", line 435, in __init__
mgr = init_dict(data, index, columns, dtype=dtype)
File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/internals/construction.py", line 254, in init_dict
return arrays_to_mgr(arrays, data_names, index, columns, dtype=dtype)
File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/internals/construction.py", line 69, in arrays_to_mgr
arrays = _homogenize(arrays, index, dtype)
File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/internals/construction.py", line 322, in _homogenize
val = sanitize_array(
File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/construction.py", line 465, in sanitize_array
subarr = construct_1d_arraylike_from_scalar(value, len(index), dtype)
File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/dtypes/cast.py", line 1461, in construct_1d_arraylike_from_scalar
subarr = np.empty(length, dtype=dtype)
TypeError: Cannot interpret '<attribute 'dtype' of 'numpy.generic' objects>' as a data type
======================================================================
ERROR: test_kde_plot (pyspark.pandas.tests.plot.test_series_plot_plotly.SeriesPlotPlotlyTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/tests/plot/test_series_plot_plotly.py", line 231, in test_kde_plot
actual = psdf.a.plot.kde(bw_method=5, ind=3)
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/plot/core.py", line 946, in kde
return self(kind="kde", bw_method=bw_method, ind=ind, **kwargs)
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/plot/core.py", line 498, in __call__
return plot_backend.plot_pandas_on_spark(plot_data, kind=kind, **kwargs)
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/plot/plotly.py", line 44, in plot_pandas_on_spark
return plot_kde(data, **kwargs)
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/plot/plotly.py", line 202, in plot_kde
pd.DataFrame(
File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/frame.py", line 435, in __init__
mgr = init_dict(data, index, columns, dtype=dtype)
File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/internals/construction.py", line 254, in init_dict
return arrays_to_mgr(arrays, data_names, index, columns, dtype=dtype)
File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/internals/construction.py", line 69, in arrays_to_mgr
arrays = _homogenize(arrays, index, dtype)
File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/internals/construction.py", line 322, in _homogenize
val = sanitize_array(
File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/construction.py", line 465, in sanitize_array
subarr = construct_1d_arraylike_from_scalar(value, len(index), dtype)
File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/dtypes/cast.py", line 1461, in construct_1d_arraylike_from_scalar
subarr = np.empty(length, dtype=dtype)
TypeError: Cannot interpret '<attribute 'dtype' of 'numpy.generic' objects>' as a data type
```
</details>
At this time, I prefer to update to **1.0.5**, I'm going to run `pyspark-pandas-slow` now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-982292481
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun edited a comment on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.0
Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-981220341
Sure, thanks for your suggestion, I'd like to update. and I added a simple test to install pandas v1.0.1 ~and run test on https://github.com/apache/spark/pull/34730 , wait for the result.~
: (, Update: pandas only publish ubuntu wheel after v1.2....we have to install many deps, otherwise it would be failed when using `pip install pandas==1.0.1`,so I just install in my local env (macos, x86, yes have the 1.0.1 wheel) and running pip install 'pandas==1.0.1' and `python/run-tests --modules=pyspark-pandas,pyspark-pandas-slow --parallelism=2 --python-executable=python3` to test it.
and looks like there were some testcase are failed:
```
======================================================================
ERROR: test_astype (pyspark.pandas.tests.data_type_ops.test_categorical_ops.CategoricalOpsTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/tests/data_type_ops/test_categorical_ops.py", line 204, in test_astype
self.assert_eq(pser.astype(int), psser.astype(int))
File "/Users/jiangyikun/spark/spark/python/pyspark/testing/pandasutils.py", line 224, in assert_eq
robj = self._to_pandas(right)
File "/Users/jiangyikun/spark/spark/python/pyspark/testing/pandasutils.py", line 245, in _to_pandas
return obj.to_pandas()
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/series.py", line 1588, in to_pandas
return self._to_pandas()
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/series.py", line 1594, in _to_pandas
return self._to_internal_pandas().copy()
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/series.py", line 6349, in _to_internal_pandas
return self._psdf._internal.to_pandas_frame[self.name]
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/utils.py", line 584, in wrapped_lazy_property
setattr(self, attr_name, fn(self))
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/internal.py", line 1049, in to_pandas_frame
pdf = sdf.toPandas()
File "/Users/jiangyikun/spark/spark/python/pyspark/sql/pandas/conversion.py", line 185, in toPandas
pdf = pd.DataFrame(columns=tmp_column_names).astype(
File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/frame.py", line 435, in __init__
mgr = init_dict(data, index, columns, dtype=dtype)
File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/internals/construction.py", line 239, in init_dict
val = construct_1d_arraylike_from_scalar(np.nan, len(index), nan_dtype)
File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/dtypes/cast.py", line 1449, in construct_1d_arraylike_from_scalar
dtype = dtype.dtype
AttributeError: type object 'object' has no attribute 'dtype'
----------------------------------------------------------------------
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] rshkv edited a comment on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.0
Posted by GitBox <gi...@apache.org>.
rshkv edited a comment on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-981126049
I noticed that `IntegralExtensionOpsTest.test_invert` fails on Pandas 1.0.0 and succeeds on 1.0.1 (error below). So maybe safer to recommend that version. Otherwise everything seems to work with 1.0.0.
<details>
<summary>Test failure</summary>
```
======================================================================
ERROR [0.404s]: test_invert (pyspark.pandas.tests.data_type_ops.test_num_ops.IntegralExtensionOpsTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/circleci/project/python/pyspark/testing/pandasutils.py", line 122, in assertPandasEqual
**kwargs
File "/opt/pyenv/versions/3.7.3/lib/python3.7/site-packages/pandas/_testing.py", line 1137, in assert_series_equal
assert_attr_equal("dtype", left, right, obj=f"Attributes of {obj}")
File "/opt/pyenv/versions/3.7.3/lib/python3.7/site-packages/pandas/_testing.py", line 772, in assert_attr_equal
raise_assert_detail(obj, msg, left_attr, right_attr)
File "/opt/pyenv/versions/3.7.3/lib/python3.7/site-packages/pandas/_testing.py", line 915, in raise_assert_detail
raise AssertionError(msg)
AssertionError: Attributes of Series are different
Attribute "dtype" are different
[left]: object
[right]: Int8
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/circleci/project/python/pyspark/pandas/tests/data_type_ops/test_num_ops.py", line 498, in test_invert
self.check_extension(~pser, ~psser)
File "/home/circleci/project/python/pyspark/pandas/tests/data_type_ops/testing_utils.py", line 248, in check_extension
self.assert_eq(left, right)
File "/home/circleci/project/python/pyspark/testing/pandasutils.py", line 223, in assert_eq
self.assertPandasEqual(lobj, robj, check_exact=check_exact)
File "/home/circleci/project/python/pyspark/testing/pandasutils.py", line 130, in assertPandasEqual
raise AssertionError(msg) from e
AssertionError: Attributes of Series are different
Attribute "dtype" are different
[left]: object
[right]: Int8
Left:
dtype: object
object
Right:
dtype: Int8
Int8
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #34717:
URL: https://github.com/apache/spark/pull/34717#discussion_r759540086
##########
File path: python/pyspark/pandas/tests/test_series.py
##########
@@ -2209,12 +2209,12 @@ def test_mad(self):
pser = pd.Series([1, 2, 3, 4], name="Koalas")
psser = ps.from_pandas(pser)
- self.assert_eq(pser.mad(), psser.mad())
+ self.assert_eq(pser.mad(), psser.mad(), almost=True)
Review comment:
Ah, got it. The result was not `DataFrame` or `Series`, then `almost=True` should be fine. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on a change in pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5
Posted by GitBox <gi...@apache.org>.
Yikun commented on a change in pull request #34717:
URL: https://github.com/apache/spark/pull/34717#discussion_r758894595
##########
File path: python/pyspark/pandas/tests/test_series.py
##########
@@ -2209,12 +2209,12 @@ def test_mad(self):
pser = pd.Series([1, 2, 3, 4], name="Koalas")
psser = ps.from_pandas(pser)
- self.assert_eq(pser.mad(), psser.mad())
+ self.assert_eq(pser.mad(), psser.mad(), almost=True)
Review comment:
Yep, I tried `self.assert_eq(pser.mad(), psser.mad(), check_exact=False)`, it failed again. Because the test.mad() result value type is a numpy.float64 scalar, it would be matched to:
https://github.com/apache/spark/blob/c38c617bcd04429e50310e4c7e0bd989c6555836/python/pyspark/testing/pandasutils.py#L237-L240
So, only `almost=True` work.
And I know your concern it's make sure the type is right in here, but not only to check value. I notice that only last check is failed in my env, so we can keep top 3 check as original, and just make last check as almost=True, and also add a note in here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun edited a comment on pull request #34717: Bump minimum pandas version to 1.0.0
Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-979659317
Just to start the discussion, by using below sql according [1], we can get the all download stat of Pandas in last 3 months.
```SQL
SELECT
file.version AS file_version,
COUNT(*) AS num_downloads,
FROM `the-psf.pypi.file_downloads`
WHERE file.project = 'pandas'
AND
-- Only query the last 3 months of history
DATE(timestamp)
BETWEEN DATE_SUB(CURRENT_DATE(), INTERVAL 3 MONTH)
AND CURRENT_DATE()
GROUP BY `file_version`
ORDER BY `num_downloads` DESC
```
Here is the Top 20 data, about 77% of the overall data, complete result can be found in [here](https://gist.github.com/Yikun/a1d8168c316966db520cf8f8a43ff0bf):
| version | number | percent
-- | -- | -- | --
1 | 0.25.3 | 35149221 | 14.28%
2 | 1.1.5 | 28722806 | 11.67%
3 | 1.3.4 | 20944236 | 8.51%
4 | 1.3.3 | 16861573 | 6.85%
5 | 0.24.2 | 13235233 | 5.38%
6 | 1.0.5 | 9201989 | 3.74%
7 | 1.3.2 | 9077326 | 3.69%
8 | 1.2.5 | 7902532 | 3.21%
9 | 1.2.4 | 5754284 | 2.34%
10 | 1.1.4 | 5710439 | 2.32%
11 | 1.1.0 | 4760847 | 1.93%
12 | 1.1.2 | 4621441 | 1.88%
13 | 1.2.3 | 4607043 | 1.87%
14 | 1.0.3 | 4601230 | 1.87%
15 | 0.23.4 | 4251044 | 1.73%
16 | 0.25.0 | 3862673 | 1.57%
17 | 1.2.1 | 2952346 | 1.20%
18 | 1.0.1 | 2690006 | 1.09%
19 | 0.22.0 | 2680710 | 1.09%
20 | 1.2.0 | 2645339 | 1.07%
21 | 0.24.1 | 2635411 | 1.07%
- There are more than 60+% users downloaded the 1.x version in last 3 months
- There are 26+% users downloaded version v0.23.2 to v1.0
[1] https://packaging.python.org/guides/analyzing-pypi-package-downloads/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.0
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-980486828
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145671/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.0
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-980486828
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145671/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-981583510
**[Test build #145720 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145720/testReport)** for PR 34717 at commit [`7b1de6d`](https://github.com/apache/spark/commit/7b1de6dba33a9313efe4eda2715ada024bfa9d0d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5
Posted by GitBox <gi...@apache.org>.
itholic commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-982192818
LGTM if remaining comments are resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-982271491
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145744/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #34717:
URL: https://github.com/apache/spark/pull/34717#discussion_r758685142
##########
File path: python/pyspark/pandas/tests/test_series.py
##########
@@ -2209,12 +2209,12 @@ def test_mad(self):
pser = pd.Series([1, 2, 3, 4], name="Koalas")
psser = ps.from_pandas(pser)
- self.assert_eq(pser.mad(), psser.mad())
+ self.assert_eq(pser.mad(), psser.mad(), almost=True)
Review comment:
Could you try with `check_exact=False` instead of `almost=True`?
If it works, it's better because it still checks the internal structure of the Series / DataFrame, whereas `almost=True` checks only values.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] rshkv commented on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.0
Posted by GitBox <gi...@apache.org>.
rshkv commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-981126049
I noticed that `IntegralExtensionOpsTest.test_invert` fails on Pandas 1.0.0 and succeeds on 1.0.1 (error below). So Maybe safer to recommend that version. Otherwise everything seems to work with 1.0.0.
<details>
<summary>Test failure</summary>
```
======================================================================
ERROR [0.404s]: test_invert (pyspark.pandas.tests.data_type_ops.test_num_ops.IntegralExtensionOpsTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/circleci/project/python/pyspark/testing/pandasutils.py", line 122, in assertPandasEqual
**kwargs
File "/opt/pyenv/versions/3.7.3/lib/python3.7/site-packages/pandas/_testing.py", line 1137, in assert_series_equal
assert_attr_equal("dtype", left, right, obj=f"Attributes of {obj}")
File "/opt/pyenv/versions/3.7.3/lib/python3.7/site-packages/pandas/_testing.py", line 772, in assert_attr_equal
raise_assert_detail(obj, msg, left_attr, right_attr)
File "/opt/pyenv/versions/3.7.3/lib/python3.7/site-packages/pandas/_testing.py", line 915, in raise_assert_detail
raise AssertionError(msg)
AssertionError: Attributes of Series are different
Attribute "dtype" are different
[left]: object
[right]: Int8
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/circleci/project/python/pyspark/pandas/tests/data_type_ops/test_num_ops.py", line 498, in test_invert
self.check_extension(~pser, ~psser)
File "/home/circleci/project/python/pyspark/pandas/tests/data_type_ops/testing_utils.py", line 248, in check_extension
self.assert_eq(left, right)
File "/home/circleci/project/python/pyspark/testing/pandasutils.py", line 223, in assert_eq
self.assertPandasEqual(lobj, robj, check_exact=check_exact)
File "/home/circleci/project/python/pyspark/testing/pandasutils.py", line 130, in assertPandasEqual
raise AssertionError(msg) from e
AssertionError: Attributes of Series are different
Attribute "dtype" are different
[left]: object
[right]: Int8
Left:
dtype: object
object
Right:
dtype: Int8
Int8
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.0
Posted by GitBox <gi...@apache.org>.
srowen commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-981182681
Yeah just require 1.0.1 for this reason
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34717: Bump minimum pandas version to 1.0.0
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-979658449
**[Test build #145645 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145645/testReport)** for PR 34717 at commit [`7986d55`](https://github.com/apache/spark/commit/7986d5591021a64ea454937e94654beab90a3042).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.0
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-980480329
**[Test build #145671 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145671/testReport)** for PR 34717 at commit [`e521b76`](https://github.com/apache/spark/commit/e521b762e0d8d078e3505480d81e01e71d21024f).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun edited a comment on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5
Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-981564669
As a conclusion in here:
- before v1.0.1: test failed due to https://github.com/apache/spark/pull/34717#issuecomment-981182023 https://github.com/apache/spark/pull/34717#issuecomment-981182023
- before v1.0.5: test failed due to https://github.com/apache/spark/pull/34717#issuecomment-981463864
- There are only a precision problem on v1.0.5, so just add `almost=True` to `assert_eq`, we can fix the testcase
So, I bump minimum pandas version to v1.0.5, the v1.0.5 is also the latest version of [Pandas verions 1.0](https://pandas.pydata.org/docs/whatsnew/index.html#version-1-0).
Ready for review. : )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun edited a comment on pull request #34717: Bump minimum pandas version to 1.0.0
Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-979659317
Just to start the discussion, by using below sql according [1], we can got the all download stat of Pandas in last 3 months.
```SQL
SELECT
file.version AS file_version,
COUNT(*) AS num_downloads,
FROM `the-psf.pypi.file_downloads`
WHERE file.project = 'pandas'
AND
-- Only query the last 3 months of history
DATE(timestamp)
BETWEEN DATE_SUB(CURRENT_DATE(), INTERVAL 3 MONTH)
AND CURRENT_DATE()
GROUP BY `file_version`
ORDER BY `num_downloads` DESC
```
Here is the Top 20 data, about 77% of the overall data, complete result can be found in [here](https://gist.github.com/Yikun/a1d8168c316966db520cf8f8a43ff0bf):
| version | number | percent
-- | -- | -- | --
1 | 0.25.3 | 35149221 | 14.28%
2 | 1.1.5 | 28722806 | 11.67%
3 | 1.3.4 | 20944236 | 8.51%
4 | 1.3.3 | 16861573 | 6.85%
5 | 0.24.2 | 13235233 | 5.38%
6 | 1.0.5 | 9201989 | 3.74%
7 | 1.3.2 | 9077326 | 3.69%
8 | 1.2.5 | 7902532 | 3.21%
9 | 1.2.4 | 5754284 | 2.34%
10 | 1.1.4 | 5710439 | 2.32%
11 | 1.1.0 | 4760847 | 1.93%
12 | 1.1.2 | 4621441 | 1.88%
13 | 1.2.3 | 4607043 | 1.87%
14 | 1.0.3 | 4601230 | 1.87%
15 | 0.23.4 | 4251044 | 1.73%
16 | 0.25.0 | 3862673 | 1.57%
17 | 1.2.1 | 2952346 | 1.20%
18 | 1.0.1 | 2690006 | 1.09%
19 | 0.22.0 | 2680710 | 1.09%
20 | 1.2.0 | 2645339 | 1.07%
21 | 0.24.1 | 2635411 | 1.07%
- There are more than 60+% users downloaded the 1.x version in last 3 months
- There are 26+% users downloaded version v0.23.2 to v1.0
[1] https://packaging.python.org/guides/analyzing-pypi-package-downloads/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34717: Bump minimum pandas version to 1.0.0
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-979658449
**[Test build #145645 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145645/testReport)** for PR 34717 at commit [`7986d55`](https://github.com/apache/spark/commit/7986d5591021a64ea454937e94654beab90a3042).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun edited a comment on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.0
Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-981220341
Sure, thanks for your suggestion, I'd like to update. and I added a simple test to install pandas v1.0.1 ~and run test on https://github.com/apache/spark/pull/34730 , wait for the result.~
: (, Update: pandas only publish ubuntu wheel after v1.2....we have to install many deps, otherwise it would be failed when using `pip install pandas==1.0.1`,so I just install in my local env (macos, x86) and running pip install 'pandas==1.0.1' and `python/run-tests --modules=pyspark-pandas,pyspark-pandas-slow --parallelism=2 --python-executable=python3` to test it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-982248418
**[Test build #145744 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145744/testReport)** for PR 34717 at commit [`054905f`](https://github.com/apache/spark/commit/054905f5074b8c543d2830e8e0500227f508a85f).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-982271491
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145744/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-982295989
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50215/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-982248418
**[Test build #145744 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145744/testReport)** for PR 34717 at commit [`054905f`](https://github.com/apache/spark/commit/054905f5074b8c543d2830e8e0500227f508a85f).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun edited a comment on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.0
Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-981463864
Complete all `pyspark-pandas` test with:
```
python/run-tests --modules=pyspark-pandas --parallelism=2 --python-executable=python3
```
Serveral test cases failed (4 cases failed due to same issue) in 1.0.1 due to `AttributeError: type object 'object' has no attribute 'dtype'` and **passed with pandas v1.0.5** (It might be fixed in https://github.com/pandas-dev/pandas/pull/34667/).
<details>
<summary>Test failure details</summary>
```
======================================================================
ERROR: test_astype (pyspark.pandas.tests.data_type_ops.test_categorical_ops.CategoricalOpsTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/tests/data_type_ops/test_categorical_ops.py", line 204, in test_astype
self.assert_eq(pser.astype(int), psser.astype(int))
File "/Users/jiangyikun/spark/spark/python/pyspark/testing/pandasutils.py", line 224, in assert_eq
robj = self._to_pandas(right)
File "/Users/jiangyikun/spark/spark/python/pyspark/testing/pandasutils.py", line 245, in _to_pandas
return obj.to_pandas()
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/series.py", line 1588, in to_pandas
return self._to_pandas()
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/series.py", line 1594, in _to_pandas
return self._to_internal_pandas().copy()
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/series.py", line 6349, in _to_internal_pandas
return self._psdf._internal.to_pandas_frame[self.name]
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/utils.py", line 584, in wrapped_lazy_property
setattr(self, attr_name, fn(self))
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/internal.py", line 1049, in to_pandas_frame
pdf = sdf.toPandas()
File "/Users/jiangyikun/spark/spark/python/pyspark/sql/pandas/conversion.py", line 185, in toPandas
pdf = pd.DataFrame(columns=tmp_column_names).astype(
File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/frame.py", line 435, in __init__
mgr = init_dict(data, index, columns, dtype=dtype)
File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/internals/construction.py", line 239, in init_dict
val = construct_1d_arraylike_from_scalar(np.nan, len(index), nan_dtype)
File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/dtypes/cast.py", line 1449, in construct_1d_arraylike_from_scalar
dtype = dtype.dtype
AttributeError: type object 'object' has no attribute 'dtype'
======================================================================
ERROR: test_read_csv (pyspark.pandas.tests.test_csv.CsvTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/tests/test_csv.py", line 151, in test_read_csv
check(usecols=[])
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/tests/test_csv.py", line 138, in check
self.assert_eq(expected, actual, almost=True)
File "/Users/jiangyikun/spark/spark/python/pyspark/testing/pandasutils.py", line 224, in assert_eq
robj = self._to_pandas(right)
File "/Users/jiangyikun/spark/spark/python/pyspark/testing/pandasutils.py", line 245, in _to_pandas
return obj.to_pandas()
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/frame.py", line 4856, in to_pandas
return self._to_pandas()
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/frame.py", line 4862, in _to_pandas
return self._internal.to_pandas_frame.copy()
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/utils.py", line 584, in wrapped_lazy_property
setattr(self, attr_name, fn(self))
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/internal.py", line 1049, in to_pandas_frame
pdf = sdf.toPandas()
File "/Users/jiangyikun/spark/spark/python/pyspark/sql/pandas/conversion.py", line 185, in toPandas
pdf = pd.DataFrame(columns=tmp_column_names).astype(
File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/frame.py", line 435, in __init__
mgr = init_dict(data, index, columns, dtype=dtype)
File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/internals/construction.py", line 239, in init_dict
val = construct_1d_arraylike_from_scalar(np.nan, len(index), nan_dtype)
File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/dtypes/cast.py", line 1449, in construct_1d_arraylike_from_scalar
dtype = dtype.dtype
AttributeError: type object 'object' has no attribute 'dtype'
======================================================================
ERROR: test_kde_plot (pyspark.pandas.tests.plot.test_frame_plot_plotly.DataFramePlotPlotlyTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/tests/plot/test_frame_plot_plotly.py", line 262, in test_kde_plot
actual = psdf.plot.kde(bw_method=5, ind=3)
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/plot/core.py", line 946, in kde
return self(kind="kde", bw_method=bw_method, ind=ind, **kwargs)
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/plot/core.py", line 498, in __call__
return plot_backend.plot_pandas_on_spark(plot_data, kind=kind, **kwargs)
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/plot/plotly.py", line 44, in plot_pandas_on_spark
return plot_kde(data, **kwargs)
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/plot/plotly.py", line 202, in plot_kde
pd.DataFrame(
File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/frame.py", line 435, in __init__
mgr = init_dict(data, index, columns, dtype=dtype)
File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/internals/construction.py", line 254, in init_dict
return arrays_to_mgr(arrays, data_names, index, columns, dtype=dtype)
File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/internals/construction.py", line 69, in arrays_to_mgr
arrays = _homogenize(arrays, index, dtype)
File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/internals/construction.py", line 322, in _homogenize
val = sanitize_array(
File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/construction.py", line 465, in sanitize_array
subarr = construct_1d_arraylike_from_scalar(value, len(index), dtype)
File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/dtypes/cast.py", line 1461, in construct_1d_arraylike_from_scalar
subarr = np.empty(length, dtype=dtype)
TypeError: Cannot interpret '<attribute 'dtype' of 'numpy.generic' objects>' as a data type
======================================================================
ERROR: test_kde_plot (pyspark.pandas.tests.plot.test_series_plot_plotly.SeriesPlotPlotlyTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/tests/plot/test_series_plot_plotly.py", line 231, in test_kde_plot
actual = psdf.a.plot.kde(bw_method=5, ind=3)
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/plot/core.py", line 946, in kde
return self(kind="kde", bw_method=bw_method, ind=ind, **kwargs)
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/plot/core.py", line 498, in __call__
return plot_backend.plot_pandas_on_spark(plot_data, kind=kind, **kwargs)
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/plot/plotly.py", line 44, in plot_pandas_on_spark
return plot_kde(data, **kwargs)
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/plot/plotly.py", line 202, in plot_kde
pd.DataFrame(
File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/frame.py", line 435, in __init__
mgr = init_dict(data, index, columns, dtype=dtype)
File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/internals/construction.py", line 254, in init_dict
return arrays_to_mgr(arrays, data_names, index, columns, dtype=dtype)
File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/internals/construction.py", line 69, in arrays_to_mgr
arrays = _homogenize(arrays, index, dtype)
File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/internals/construction.py", line 322, in _homogenize
val = sanitize_array(
File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/construction.py", line 465, in sanitize_array
subarr = construct_1d_arraylike_from_scalar(value, len(index), dtype)
File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/dtypes/cast.py", line 1461, in construct_1d_arraylike_from_scalar
subarr = np.empty(length, dtype=dtype)
TypeError: Cannot interpret '<attribute 'dtype' of 'numpy.generic' objects>' as a data type
```
</details>
At this time, I prefer to update to **1.0.5**, I'm going to run `pyspark-pandas-slow` now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.0
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-980483121
**[Test build #145671 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145671/testReport)** for PR 34717 at commit [`e521b76`](https://github.com/apache/spark/commit/e521b762e0d8d078e3505480d81e01e71d21024f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.0
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-980492640
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50141/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-981627078
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145720/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-982295989
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50215/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-981734229
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50189/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-982270701
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50215/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-981666882
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50189/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #34717:
URL: https://github.com/apache/spark/pull/34717
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34717: Bump minimum pandas version to 1.0.0
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-979676293
**[Test build #145645 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145645/testReport)** for PR 34717 at commit [`7986d55`](https://github.com/apache/spark/commit/7986d5591021a64ea454937e94654beab90a3042).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34717: Bump minimum pandas version to 1.0.0
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-979681958
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145645/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.0
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-981212899
Thanks @Yikun, how do you think about bumping to `1.0.1`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.0
Posted by GitBox <gi...@apache.org>.
Yikun commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-981220341
Sure, thanks for your suggestion, I'd like to update. and I added a simple test to install pandas v1.0.1 and run test on https://github.com/apache/spark/pull/34730 , wait for the result.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-981583510
**[Test build #145720 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145720/testReport)** for PR 34717 at commit [`7b1de6d`](https://github.com/apache/spark/commit/7b1de6dba33a9313efe4eda2715ada024bfa9d0d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-981705646
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50189/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun edited a comment on pull request #34717: Bump minimum pandas version to 1.0.0
Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-979659317
Just to start the discussion:
Here is top 20 download from pypi stat:
By using below sql according [1], we can got the all download stat of Pandas in last 3 months.
```SQL
SELECT
file.version AS file_version,
COUNT(*) AS num_downloads,
FROM `the-psf.pypi.file_downloads`
WHERE file.project = 'pandas'
AND
-- Only query the last 3 months of history
DATE(timestamp)
BETWEEN DATE_SUB(CURRENT_DATE(), INTERVAL 3 MONTH)
AND CURRENT_DATE()
GROUP BY `file_version`
ORDER BY `num_downloads` DESC
```
Here is the Top 20 data, about 77% of the overall data:
| version | number | percent
-- | -- | -- | --
1 | 0.25.3 | 35149221 | 14.28%
2 | 1.1.5 | 28722806 | 11.67%
3 | 1.3.4 | 20944236 | 8.51%
4 | 1.3.3 | 16861573 | 6.85%
5 | 0.24.2 | 13235233 | 5.38%
6 | 1.0.5 | 9201989 | 3.74%
7 | 1.3.2 | 9077326 | 3.69%
8 | 1.2.5 | 7902532 | 3.21%
9 | 1.2.4 | 5754284 | 2.34%
10 | 1.1.4 | 5710439 | 2.32%
11 | 1.1.0 | 4760847 | 1.93%
12 | 1.1.2 | 4621441 | 1.88%
13 | 1.2.3 | 4607043 | 1.87%
14 | 1.0.3 | 4601230 | 1.87%
15 | 0.23.4 | 4251044 | 1.73%
16 | 0.25.0 | 3862673 | 1.57%
17 | 1.2.1 | 2952346 | 1.20%
18 | 1.0.1 | 2690006 | 1.09%
19 | 0.22.0 | 2680710 | 1.09%
20 | 1.2.0 | 2645339 | 1.07%
21 | 0.24.1 | 2635411 | 1.07%
[1] https://packaging.python.org/guides/analyzing-pypi-package-downloads/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #34717: Bump minimum pandas version to 1.0.0
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-979660513
Let's also update https://github.com/apache/spark/blob/master/python/docs/source/getting_started/install.rst#dependencies
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #34717: Bump minimum pandas version to 1.0.0
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-979660278
cc @ueshin @xinrong-databricks @itholic FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34717: Bump minimum pandas version to 1.0.0
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-979699593
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50117/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-982260719
**[Test build #145744 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145744/testReport)** for PR 34717 at commit [`054905f`](https://github.com/apache/spark/commit/054905f5074b8c543d2830e8e0500227f508a85f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34717: Bump minimum pandas version to 1.0.0
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-979699593
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50117/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun edited a comment on pull request #34717: Bump minimum pandas version to 1.0.0
Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-979659317
Just to start the discussion, by using below sql according [1], we can got the all download stat of Pandas in last 3 months.
```SQL
SELECT
file.version AS file_version,
COUNT(*) AS num_downloads,
FROM `the-psf.pypi.file_downloads`
WHERE file.project = 'pandas'
AND
-- Only query the last 3 months of history
DATE(timestamp)
BETWEEN DATE_SUB(CURRENT_DATE(), INTERVAL 3 MONTH)
AND CURRENT_DATE()
GROUP BY `file_version`
ORDER BY `num_downloads` DESC
```
Here is the Top 20 data, about 77% of the overall data, complete result can be found in [here](https://gist.github.com/Yikun/a1d8168c316966db520cf8f8a43ff0bf):
| version | number | percent
-- | -- | -- | --
1 | 0.25.3 | 35149221 | 14.28%
2 | 1.1.5 | 28722806 | 11.67%
3 | 1.3.4 | 20944236 | 8.51%
4 | 1.3.3 | 16861573 | 6.85%
5 | 0.24.2 | 13235233 | 5.38%
6 | 1.0.5 | 9201989 | 3.74%
7 | 1.3.2 | 9077326 | 3.69%
8 | 1.2.5 | 7902532 | 3.21%
9 | 1.2.4 | 5754284 | 2.34%
10 | 1.1.4 | 5710439 | 2.32%
11 | 1.1.0 | 4760847 | 1.93%
12 | 1.1.2 | 4621441 | 1.88%
13 | 1.2.3 | 4607043 | 1.87%
14 | 1.0.3 | 4601230 | 1.87%
15 | 0.23.4 | 4251044 | 1.73%
16 | 0.25.0 | 3862673 | 1.57%
17 | 1.2.1 | 2952346 | 1.20%
18 | 1.0.1 | 2690006 | 1.09%
19 | 0.22.0 | 2680710 | 1.09%
20 | 1.2.0 | 2645339 | 1.07%
21 | 0.24.1 | 2635411 | 1.07%
[1] https://packaging.python.org/guides/analyzing-pypi-package-downloads/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34717: Bump minimum pandas version to 1.0.0
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-979681958
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145645/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on pull request #34717: Bump minimum pandas version to 1.0.0
Posted by GitBox <gi...@apache.org>.
Yikun commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-979659317
Here is top 20 download from pypi stat:
By using below way according
```SQL
SELECT
file.version AS file_version,
COUNT(*) AS num_downloads,
FROM `the-psf.pypi.file_downloads`
WHERE file.project = 'pandas'
AND
-- Only query the last 3 months of history
DATE(timestamp)
BETWEEN DATE_SUB(CURRENT_DATE(), INTERVAL 3 MONTH)
AND CURRENT_DATE()
GROUP BY `file_version`
ORDER BY `num_downloads` DESC
```
| version | number | percent
-- | -- | -- | --
1 | 0.25.3 | 35149221 | 14.28%
2 | 1.1.5 | 28722806 | 11.67%
3 | 1.3.4 | 20944236 | 8.51%
4 | 1.3.3 | 16861573 | 6.85%
5 | 0.24.2 | 13235233 | 5.38%
6 | 1.0.5 | 9201989 | 3.74%
7 | 1.3.2 | 9077326 | 3.69%
8 | 1.2.5 | 7902532 | 3.21%
9 | 1.2.4 | 5754284 | 2.34%
10 | 1.1.4 | 5710439 | 2.32%
11 | 1.1.0 | 4760847 | 1.93%
12 | 1.1.2 | 4621441 | 1.88%
13 | 1.2.3 | 4607043 | 1.87%
14 | 1.0.3 | 4601230 | 1.87%
15 | 0.23.4 | 4251044 | 1.73%
16 | 0.25.0 | 3862673 | 1.57%
17 | 1.2.1 | 2952346 | 1.20%
18 | 1.0.1 | 2690006 | 1.09%
19 | 0.22.0 | 2680710 | 1.09%
20 | 1.2.0 | 2645339 | 1.07%
21 | 0.24.1 | 2635411 | 1.07%
[1] https://packaging.python.org/guides/analyzing-pypi-package-downloads/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-981734229
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50189/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.0
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-980492640
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50141/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on a change in pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5
Posted by GitBox <gi...@apache.org>.
Yikun commented on a change in pull request #34717:
URL: https://github.com/apache/spark/pull/34717#discussion_r758898358
##########
File path: python/docs/source/user_guide/sql/arrow_pandas.rst
##########
@@ -387,7 +387,7 @@ working with timestamps in ``pandas_udf``\s to get the best performance, see
Recommended Pandas and PyArrow Versions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-For usage with pyspark.sql, the minimum supported versions of Pandas is 0.23.2 and PyArrow is 1.0.0.
+For usage with pyspark.sql, the minimum supported versions of Pandas is 1.0.5 and PyArrow is 1.0.0.
Review comment:
How about:
For usage with pyspark.sql, the minimum supported versions of Pandas is **1.0.5** and PyArrow is 1.0.0. **Lower versions (such as there are some known issues under with v1.0.0, v1.0.1, see more in [link](https://github.com/apache/spark/pull/34717)) or** higher versions may be used, however, compatibility and data correctness can not be guaranteed and should be verified by the user.
Maybe need more suggestion from native speaker. T_T, and if it's necessary we could do it in next commits in this PR or followup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.0
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-980484750
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50141/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun edited a comment on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.0
Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-981220341
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun edited a comment on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5
Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-981564669
As a conclusion in here:
- before v1.0.1: test failed due to https://github.com/apache/spark/pull/34717#issuecomment-981182023 https://github.com/apache/spark/pull/34717#issuecomment-981182023
- before v1.0.5: test failed due to https://github.com/apache/spark/pull/34717#issuecomment-981463864
- v1.0.5: There are only a precision problem, so I just added `almost=True` to `assert_eq` to fix the testcase
So, I bump minimum pandas version to v1.0.5, the v1.0.5 is also the latest version of [Pandas verions 1.0](https://pandas.pydata.org/docs/whatsnew/index.html#version-1-0).
Ready for review. : )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on a change in pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5
Posted by GitBox <gi...@apache.org>.
Yikun commented on a change in pull request #34717:
URL: https://github.com/apache/spark/pull/34717#discussion_r758894595
##########
File path: python/pyspark/pandas/tests/test_series.py
##########
@@ -2209,12 +2209,12 @@ def test_mad(self):
pser = pd.Series([1, 2, 3, 4], name="Koalas")
psser = ps.from_pandas(pser)
- self.assert_eq(pser.mad(), psser.mad())
+ self.assert_eq(pser.mad(), psser.mad(), almost=True)
Review comment:
Yep, I tried `self.assert_eq(pser.mad(), psser.mad(), check_exact=False)`, it failed again. Because the test.mad() result value type is a numpy.float64 scalar, it would be matched to:
https://github.com/apache/spark/blob/c38c617bcd04429e50310e4c7e0bd989c6555836/python/pyspark/testing/pandasutils.py#L237-L240
So, only `almost=True` work.
And I know your concern it's make sure the type is right in here, but not only to check value. I notice that only last check is failed in my env, so we can keep top 3 check as original, and just make last check as almost=True.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.0
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-980480329
**[Test build #145671 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145671/testReport)** for PR 34717 at commit [`e521b76`](https://github.com/apache/spark/commit/e521b762e0d8d078e3505480d81e01e71d21024f).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.0
Posted by GitBox <gi...@apache.org>.
itholic commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-981182023
> I noticed that `IntegralExtensionOpsTest.test_invert` fails on Pandas 1.0.0 and succeeds on 1.0.1 (error below). So maybe safer to recommend that version. Otherwise everything seems to work with 1.0.0.
Yeah, it seems the bug in pandas 1.0.0.
- pandas 1.0.0
```python
>>> pser = pd.Series([1, 2, 3, None], dtype="Int8")
>>> pser
0 1
1 2
2 3
3 <NA>
dtype: Int8
>>> ~pser
0 -2
1 -3
2 -4
3 <NA>
dtype: object # this should've been `Int8`
```
Resolved in pandas 1.0.1.
- pandas 1.0.1
```python
>>> pser = pd.Series([1, 2, 3, None], dtype="Int8")
>>> pser
0 1
1 2
2 3
3 <NA>
dtype: Int8
>>> ~pser
0 -2
1 -3
2 -4
3 <NA>
dtype: Int8
```
For addressing this,
1. manually separate this test for pandas 1.0.0.
2. set the minimum pandas to version 1.0.1.
Not sure which way is better, but I think we can just go with `2` if there is no reason to stick with the 1.0.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun edited a comment on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.0
Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-981463864
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun edited a comment on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.0
Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-981463864
Complete all `pyspark-pandas` test with:
```
python/run-tests --modules=pyspark-pandas --parallelism=2 --python-executable=python3
```
Serveral test cases failed (5 failed due to same issue) in 1.0.1 due to `AttributeError: type object 'object' has no attribute 'dtype'` and **passed with pandas v1.0.5** (It might be fixed in https://github.com/pandas-dev/pandas/pull/34667/).
<details>
<summary>Test failure details</summary>
```
======================================================================
ERROR: test_astype (pyspark.pandas.tests.data_type_ops.test_categorical_ops.CategoricalOpsTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/tests/data_type_ops/test_categorical_ops.py", line 204, in test_astype
self.assert_eq(pser.astype(int), psser.astype(int))
File "/Users/jiangyikun/spark/spark/python/pyspark/testing/pandasutils.py", line 224, in assert_eq
robj = self._to_pandas(right)
File "/Users/jiangyikun/spark/spark/python/pyspark/testing/pandasutils.py", line 245, in _to_pandas
return obj.to_pandas()
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/series.py", line 1588, in to_pandas
return self._to_pandas()
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/series.py", line 1594, in _to_pandas
return self._to_internal_pandas().copy()
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/series.py", line 6349, in _to_internal_pandas
return self._psdf._internal.to_pandas_frame[self.name]
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/utils.py", line 584, in wrapped_lazy_property
setattr(self, attr_name, fn(self))
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/internal.py", line 1049, in to_pandas_frame
pdf = sdf.toPandas()
File "/Users/jiangyikun/spark/spark/python/pyspark/sql/pandas/conversion.py", line 185, in toPandas
pdf = pd.DataFrame(columns=tmp_column_names).astype(
File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/frame.py", line 435, in __init__
mgr = init_dict(data, index, columns, dtype=dtype)
File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/internals/construction.py", line 239, in init_dict
val = construct_1d_arraylike_from_scalar(np.nan, len(index), nan_dtype)
File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/dtypes/cast.py", line 1449, in construct_1d_arraylike_from_scalar
dtype = dtype.dtype
AttributeError: type object 'object' has no attribute 'dtype'
======================================================================
ERROR: test_read_csv (pyspark.pandas.tests.test_csv.CsvTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/tests/test_csv.py", line 151, in test_read_csv
check(usecols=[])
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/tests/test_csv.py", line 138, in check
self.assert_eq(expected, actual, almost=True)
File "/Users/jiangyikun/spark/spark/python/pyspark/testing/pandasutils.py", line 224, in assert_eq
robj = self._to_pandas(right)
File "/Users/jiangyikun/spark/spark/python/pyspark/testing/pandasutils.py", line 245, in _to_pandas
return obj.to_pandas()
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/frame.py", line 4856, in to_pandas
return self._to_pandas()
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/frame.py", line 4862, in _to_pandas
return self._internal.to_pandas_frame.copy()
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/utils.py", line 584, in wrapped_lazy_property
setattr(self, attr_name, fn(self))
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/internal.py", line 1049, in to_pandas_frame
pdf = sdf.toPandas()
File "/Users/jiangyikun/spark/spark/python/pyspark/sql/pandas/conversion.py", line 185, in toPandas
pdf = pd.DataFrame(columns=tmp_column_names).astype(
File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/frame.py", line 435, in __init__
mgr = init_dict(data, index, columns, dtype=dtype)
File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/internals/construction.py", line 239, in init_dict
val = construct_1d_arraylike_from_scalar(np.nan, len(index), nan_dtype)
File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/dtypes/cast.py", line 1449, in construct_1d_arraylike_from_scalar
dtype = dtype.dtype
AttributeError: type object 'object' has no attribute 'dtype'
======================================================================
ERROR: test_kde_plot (pyspark.pandas.tests.plot.test_frame_plot_plotly.DataFramePlotPlotlyTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/tests/plot/test_frame_plot_plotly.py", line 262, in test_kde_plot
actual = psdf.plot.kde(bw_method=5, ind=3)
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/plot/core.py", line 946, in kde
return self(kind="kde", bw_method=bw_method, ind=ind, **kwargs)
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/plot/core.py", line 498, in __call__
return plot_backend.plot_pandas_on_spark(plot_data, kind=kind, **kwargs)
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/plot/plotly.py", line 44, in plot_pandas_on_spark
return plot_kde(data, **kwargs)
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/plot/plotly.py", line 202, in plot_kde
pd.DataFrame(
File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/frame.py", line 435, in __init__
mgr = init_dict(data, index, columns, dtype=dtype)
File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/internals/construction.py", line 254, in init_dict
return arrays_to_mgr(arrays, data_names, index, columns, dtype=dtype)
File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/internals/construction.py", line 69, in arrays_to_mgr
arrays = _homogenize(arrays, index, dtype)
File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/internals/construction.py", line 322, in _homogenize
val = sanitize_array(
File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/construction.py", line 465, in sanitize_array
subarr = construct_1d_arraylike_from_scalar(value, len(index), dtype)
File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/dtypes/cast.py", line 1461, in construct_1d_arraylike_from_scalar
subarr = np.empty(length, dtype=dtype)
TypeError: Cannot interpret '<attribute 'dtype' of 'numpy.generic' objects>' as a data type
======================================================================
ERROR: test_kde_plot (pyspark.pandas.tests.plot.test_series_plot_plotly.SeriesPlotPlotlyTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/tests/plot/test_series_plot_plotly.py", line 231, in test_kde_plot
actual = psdf.a.plot.kde(bw_method=5, ind=3)
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/plot/core.py", line 946, in kde
return self(kind="kde", bw_method=bw_method, ind=ind, **kwargs)
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/plot/core.py", line 498, in __call__
return plot_backend.plot_pandas_on_spark(plot_data, kind=kind, **kwargs)
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/plot/plotly.py", line 44, in plot_pandas_on_spark
return plot_kde(data, **kwargs)
File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/plot/plotly.py", line 202, in plot_kde
pd.DataFrame(
File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/frame.py", line 435, in __init__
mgr = init_dict(data, index, columns, dtype=dtype)
File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/internals/construction.py", line 254, in init_dict
return arrays_to_mgr(arrays, data_names, index, columns, dtype=dtype)
File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/internals/construction.py", line 69, in arrays_to_mgr
arrays = _homogenize(arrays, index, dtype)
File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/internals/construction.py", line 322, in _homogenize
val = sanitize_array(
File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/construction.py", line 465, in sanitize_array
subarr = construct_1d_arraylike_from_scalar(value, len(index), dtype)
File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/dtypes/cast.py", line 1461, in construct_1d_arraylike_from_scalar
subarr = np.empty(length, dtype=dtype)
TypeError: Cannot interpret '<attribute 'dtype' of 'numpy.generic' objects>' as a data type
```
</details>
At this time, I prefer to update to **1.0.5**, I'm going to run `pyspark-pandas-slow` now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #34717:
URL: https://github.com/apache/spark/pull/34717#discussion_r758685142
##########
File path: python/pyspark/pandas/tests/test_series.py
##########
@@ -2209,12 +2209,12 @@ def test_mad(self):
pser = pd.Series([1, 2, 3, 4], name="Koalas")
psser = ps.from_pandas(pser)
- self.assert_eq(pser.mad(), psser.mad())
+ self.assert_eq(pser.mad(), psser.mad(), almost=True)
Review comment:
Could you try with `check_exact=False` instead of `almost=True`?
If it works, it's better because it still checks the internal structure of the Series / DataFrame; otherwise `almost=True` checks only values.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on a change in pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5
Posted by GitBox <gi...@apache.org>.
Yikun commented on a change in pull request #34717:
URL: https://github.com/apache/spark/pull/34717#discussion_r758895254
##########
File path: python/pyspark/pandas/tests/test_series.py
##########
@@ -2209,12 +2209,12 @@ def test_mad(self):
pser = pd.Series([1, 2, 3, 4], name="Koalas")
psser = ps.from_pandas(pser)
- self.assert_eq(pser.mad(), psser.mad())
+ self.assert_eq(pser.mad(), psser.mad(), almost=True)
Review comment:
And one more thing, this test case failed in my env always, even the latest pandas v1.3.4, so looks like a precision issue cross platfroms.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on a change in pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5
Posted by GitBox <gi...@apache.org>.
Yikun commented on a change in pull request #34717:
URL: https://github.com/apache/spark/pull/34717#discussion_r758898358
##########
File path: python/docs/source/user_guide/sql/arrow_pandas.rst
##########
@@ -387,7 +387,7 @@ working with timestamps in ``pandas_udf``\s to get the best performance, see
Recommended Pandas and PyArrow Versions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-For usage with pyspark.sql, the minimum supported versions of Pandas is 0.23.2 and PyArrow is 1.0.0.
+For usage with pyspark.sql, the minimum supported versions of Pandas is 1.0.5 and PyArrow is 1.0.0.
Review comment:
How about?
For usage with pyspark.sql, the minimum supported versions of Pandas is 1.0.5 and PyArrow is 1.0.0. Lower versions (such as there are some known issue under with v1.0.0, v1.0.1, see more in [link](https://github.com/apache/spark/pull/34717)) or higher versions may be used, however, compatibility and data correctness can not be guaranteed and should be verified by the user.
Maybe need more suggestion from native speaker. T_T
##########
File path: python/docs/source/user_guide/sql/arrow_pandas.rst
##########
@@ -387,7 +387,7 @@ working with timestamps in ``pandas_udf``\s to get the best performance, see
Recommended Pandas and PyArrow Versions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-For usage with pyspark.sql, the minimum supported versions of Pandas is 0.23.2 and PyArrow is 1.0.0.
+For usage with pyspark.sql, the minimum supported versions of Pandas is 1.0.5 and PyArrow is 1.0.0.
Review comment:
How about?
For usage with pyspark.sql, the minimum supported versions of Pandas is 1.0.5 and PyArrow is 1.0.0. Lower versions (such as there are some known issues under with v1.0.0, v1.0.1, see more in [link](https://github.com/apache/spark/pull/34717)) or higher versions may be used, however, compatibility and data correctness can not be guaranteed and should be verified by the user.
Maybe need more suggestion from native speaker. T_T
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun edited a comment on pull request #34717: Bump minimum pandas version to 1.0.0
Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-979659317
Just to start the discussion:
Here is top 20 download from pypi stat:
By using below sql according [1], we can got the top 20 download version of Pandas in last 3 months.
```SQL
SELECT
file.version AS file_version,
COUNT(*) AS num_downloads,
FROM `the-psf.pypi.file_downloads`
WHERE file.project = 'pandas'
AND
-- Only query the last 3 months of history
DATE(timestamp)
BETWEEN DATE_SUB(CURRENT_DATE(), INTERVAL 3 MONTH)
AND CURRENT_DATE()
GROUP BY `file_version`
ORDER BY `num_downloads` DESC
```
| version | number | percent
-- | -- | -- | --
1 | 0.25.3 | 35149221 | 14.28%
2 | 1.1.5 | 28722806 | 11.67%
3 | 1.3.4 | 20944236 | 8.51%
4 | 1.3.3 | 16861573 | 6.85%
5 | 0.24.2 | 13235233 | 5.38%
6 | 1.0.5 | 9201989 | 3.74%
7 | 1.3.2 | 9077326 | 3.69%
8 | 1.2.5 | 7902532 | 3.21%
9 | 1.2.4 | 5754284 | 2.34%
10 | 1.1.4 | 5710439 | 2.32%
11 | 1.1.0 | 4760847 | 1.93%
12 | 1.1.2 | 4621441 | 1.88%
13 | 1.2.3 | 4607043 | 1.87%
14 | 1.0.3 | 4601230 | 1.87%
15 | 0.23.4 | 4251044 | 1.73%
16 | 0.25.0 | 3862673 | 1.57%
17 | 1.2.1 | 2952346 | 1.20%
18 | 1.0.1 | 2690006 | 1.09%
19 | 0.22.0 | 2680710 | 1.09%
20 | 1.2.0 | 2645339 | 1.07%
21 | 0.24.1 | 2635411 | 1.07%
[1] https://packaging.python.org/guides/analyzing-pypi-package-downloads/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun edited a comment on pull request #34717: Bump minimum pandas version to 1.0.0
Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-979659317
Just to start the discussion:
Here is top 20 download from pypi stat:
By using below way according [1]
```SQL
SELECT
file.version AS file_version,
COUNT(*) AS num_downloads,
FROM `the-psf.pypi.file_downloads`
WHERE file.project = 'pandas'
AND
-- Only query the last 3 months of history
DATE(timestamp)
BETWEEN DATE_SUB(CURRENT_DATE(), INTERVAL 3 MONTH)
AND CURRENT_DATE()
GROUP BY `file_version`
ORDER BY `num_downloads` DESC
```
| version | number | percent
-- | -- | -- | --
1 | 0.25.3 | 35149221 | 14.28%
2 | 1.1.5 | 28722806 | 11.67%
3 | 1.3.4 | 20944236 | 8.51%
4 | 1.3.3 | 16861573 | 6.85%
5 | 0.24.2 | 13235233 | 5.38%
6 | 1.0.5 | 9201989 | 3.74%
7 | 1.3.2 | 9077326 | 3.69%
8 | 1.2.5 | 7902532 | 3.21%
9 | 1.2.4 | 5754284 | 2.34%
10 | 1.1.4 | 5710439 | 2.32%
11 | 1.1.0 | 4760847 | 1.93%
12 | 1.1.2 | 4621441 | 1.88%
13 | 1.2.3 | 4607043 | 1.87%
14 | 1.0.3 | 4601230 | 1.87%
15 | 0.23.4 | 4251044 | 1.73%
16 | 0.25.0 | 3862673 | 1.57%
17 | 1.2.1 | 2952346 | 1.20%
18 | 1.0.1 | 2690006 | 1.09%
19 | 0.22.0 | 2680710 | 1.09%
20 | 1.2.0 | 2645339 | 1.07%
21 | 0.24.1 | 2635411 | 1.07%
[1] https://packaging.python.org/guides/analyzing-pypi-package-downloads/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun edited a comment on pull request #34717: Bump minimum pandas version to 1.0.0
Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-979659317
Here is top 20 download from pypi stat:
By using below way according [1]
```SQL
SELECT
file.version AS file_version,
COUNT(*) AS num_downloads,
FROM `the-psf.pypi.file_downloads`
WHERE file.project = 'pandas'
AND
-- Only query the last 3 months of history
DATE(timestamp)
BETWEEN DATE_SUB(CURRENT_DATE(), INTERVAL 3 MONTH)
AND CURRENT_DATE()
GROUP BY `file_version`
ORDER BY `num_downloads` DESC
```
| version | number | percent
-- | -- | -- | --
1 | 0.25.3 | 35149221 | 14.28%
2 | 1.1.5 | 28722806 | 11.67%
3 | 1.3.4 | 20944236 | 8.51%
4 | 1.3.3 | 16861573 | 6.85%
5 | 0.24.2 | 13235233 | 5.38%
6 | 1.0.5 | 9201989 | 3.74%
7 | 1.3.2 | 9077326 | 3.69%
8 | 1.2.5 | 7902532 | 3.21%
9 | 1.2.4 | 5754284 | 2.34%
10 | 1.1.4 | 5710439 | 2.32%
11 | 1.1.0 | 4760847 | 1.93%
12 | 1.1.2 | 4621441 | 1.88%
13 | 1.2.3 | 4607043 | 1.87%
14 | 1.0.3 | 4601230 | 1.87%
15 | 0.23.4 | 4251044 | 1.73%
16 | 0.25.0 | 3862673 | 1.57%
17 | 1.2.1 | 2952346 | 1.20%
18 | 1.0.1 | 2690006 | 1.09%
19 | 0.22.0 | 2680710 | 1.09%
20 | 1.2.0 | 2645339 | 1.07%
21 | 0.24.1 | 2635411 | 1.07%
[1] https://packaging.python.org/guides/analyzing-pypi-package-downloads/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34717: Bump minimum pandas version to 1.0.0
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-979696936
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50117/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-981608086
**[Test build #145720 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145720/testReport)** for PR 34717 at commit [`7b1de6d`](https://github.com/apache/spark/commit/7b1de6dba33a9313efe4eda2715ada024bfa9d0d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-981627078
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145720/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org