You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/11/26 03:18:11 UTC

[GitHub] [spark] Yikun opened a new pull request #34717: Bump minimum pandas version to 1.0.0

Yikun opened a new pull request #34717:
URL: https://github.com/apache/spark/pull/34717


   ### What changes were proposed in this pull request?
   Bump minimum pandas version to 1.0.0
   
   ### Why are the changes needed?
   To be done
   
   
   ### Does this PR introduce _any_ user-facing change?
   To be done
   
   ### How was this patch tested?
   To be done


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Yikun edited a comment on pull request #34717: Bump minimum pandas version to 1.0.0

Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-979659317


   Just to start the discussion, by using below sql according [1], we can got the all download stat of Pandas in last 3 months.
   ```SQL
   SELECT
     file.version AS file_version,
     COUNT(*) AS num_downloads,
   FROM `the-psf.pypi.file_downloads`
   WHERE file.project = 'pandas'
   AND 
     -- Only query the last 3 months of history
     DATE(timestamp)
       BETWEEN DATE_SUB(CURRENT_DATE(), INTERVAL 3 MONTH)
       AND CURRENT_DATE()
   GROUP BY `file_version`
   ORDER BY `num_downloads` DESC
   ```
   
   Here is the Top 20 data, about 77% of the overall data:
     | version | number | percent
   -- | -- | -- | --
   1 | 0.25.3 | 35149221 | 14.28%
   2 | 1.1.5 | 28722806 | 11.67%
   3 | 1.3.4 | 20944236 | 8.51%
   4 | 1.3.3 | 16861573 | 6.85%
   5 | 0.24.2 | 13235233 | 5.38%
   6 | 1.0.5 | 9201989 | 3.74%
   7 | 1.3.2 | 9077326 | 3.69%
   8 | 1.2.5 | 7902532 | 3.21%
   9 | 1.2.4 | 5754284 | 2.34%
   10 | 1.1.4 | 5710439 | 2.32%
   11 | 1.1.0 | 4760847 | 1.93%
   12 | 1.1.2 | 4621441 | 1.88%
   13 | 1.2.3 | 4607043 | 1.87%
   14 | 1.0.3 | 4601230 | 1.87%
   15 | 0.23.4 | 4251044 | 1.73%
   16 | 0.25.0 | 3862673 | 1.57%
   17 | 1.2.1 | 2952346 | 1.20%
   18 | 1.0.1 | 2690006 | 1.09%
   19 | 0.22.0 | 2680710 | 1.09%
   20 | 1.2.0 | 2645339 | 1.07%
   21 | 0.24.1 | 2635411 | 1.07%
   
   [1] https://packaging.python.org/guides/analyzing-pypi-package-downloads/


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Yikun edited a comment on pull request #34717: Bump minimum pandas version to 1.0.0

Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-979659317


   Just to start the discussion, by using below sql according [1], we can got the all download stat of Pandas in last 3 months.
   ```SQL
   SELECT
     file.version AS file_version,
     COUNT(*) AS num_downloads,
   FROM `the-psf.pypi.file_downloads`
   WHERE file.project = 'pandas'
   AND 
     -- Only query the last 3 months of history
     DATE(timestamp)
       BETWEEN DATE_SUB(CURRENT_DATE(), INTERVAL 3 MONTH)
       AND CURRENT_DATE()
   GROUP BY `file_version`
   ORDER BY `num_downloads` DESC
   ```
   
   Here is the Top 20 data, about 77% of the overall data, complete result can be found in [here](https://gist.github.com/Yikun/a1d8168c316966db520cf8f8a43ff0bf):
     | version | number | percent
   -- | -- | -- | --
   1 | 0.25.3 | 35149221 | 14.28%
   2 | 1.1.5 | 28722806 | 11.67%
   3 | 1.3.4 | 20944236 | 8.51%
   4 | 1.3.3 | 16861573 | 6.85%
   5 | 0.24.2 | 13235233 | 5.38%
   6 | 1.0.5 | 9201989 | 3.74%
   7 | 1.3.2 | 9077326 | 3.69%
   8 | 1.2.5 | 7902532 | 3.21%
   9 | 1.2.4 | 5754284 | 2.34%
   10 | 1.1.4 | 5710439 | 2.32%
   11 | 1.1.0 | 4760847 | 1.93%
   12 | 1.1.2 | 4621441 | 1.88%
   13 | 1.2.3 | 4607043 | 1.87%
   14 | 1.0.3 | 4601230 | 1.87%
   15 | 0.23.4 | 4251044 | 1.73%
   16 | 0.25.0 | 3862673 | 1.57%
   17 | 1.2.1 | 2952346 | 1.20%
   18 | 1.0.1 | 2690006 | 1.09%
   19 | 0.22.0 | 2680710 | 1.09%
   20 | 1.2.0 | 2645339 | 1.07%
   21 | 0.24.1 | 2635411 | 1.07%
   
   [1] https://packaging.python.org/guides/analyzing-pypi-package-downloads/


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Yikun commented on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5

Posted by GitBox <gi...@apache.org>.
Yikun commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-981564669


   As a conclusion in here:
   - before v1.0.1: test failed due to https://github.com/apache/spark/pull/34717#issuecomment-981182023 https://github.com/apache/spark/pull/34717#issuecomment-981182023
   - before v1.0.5: test failed due to https://github.com/apache/spark/pull/34717#issuecomment-981463864
   - There are only a precision problem on v1.0.5, so just add `almost=True` to `assert_eq`, we can fix the testcase
   
   So, I bump minimum pandas version to v1.0.5, the v1.0.5 is also the latest version of [Pandas verions 1.0](https://pandas.pydata.org/docs/whatsnew/index.html#version-1-0).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-982288646


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50215/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34717: Bump minimum pandas version to 1.0.0

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-979683584


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50117/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Yikun edited a comment on pull request #34717: Bump minimum pandas version to 1.0.0

Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-979659317


   Just to start the discussion, by using below sql according [1], we can got the all download stat of Pandas in last 3 months.
   ```SQL
   SELECT
     file.version AS file_version,
     COUNT(*) AS num_downloads,
   FROM `the-psf.pypi.file_downloads`
   WHERE file.project = 'pandas'
   AND 
     -- Only query the last 3 months of history
     DATE(timestamp)
       BETWEEN DATE_SUB(CURRENT_DATE(), INTERVAL 3 MONTH)
       AND CURRENT_DATE()
   GROUP BY `file_version`
   ORDER BY `num_downloads` DESC
   ```
   
   Here is the Top 20 data, about 77% of the overall data, complete result can be found in [here](https://gist.github.com/Yikun/a1d8168c316966db520cf8f8a43ff0bf):
     | version | number | percent
   -- | -- | -- | --
   1 | 0.25.3 | 35149221 | 14.28%
   2 | 1.1.5 | 28722806 | 11.67%
   3 | 1.3.4 | 20944236 | 8.51%
   4 | 1.3.3 | 16861573 | 6.85%
   5 | 0.24.2 | 13235233 | 5.38%
   6 | 1.0.5 | 9201989 | 3.74%
   7 | 1.3.2 | 9077326 | 3.69%
   8 | 1.2.5 | 7902532 | 3.21%
   9 | 1.2.4 | 5754284 | 2.34%
   10 | 1.1.4 | 5710439 | 2.32%
   11 | 1.1.0 | 4760847 | 1.93%
   12 | 1.1.2 | 4621441 | 1.88%
   13 | 1.2.3 | 4607043 | 1.87%
   14 | 1.0.3 | 4601230 | 1.87%
   15 | 0.23.4 | 4251044 | 1.73%
   16 | 0.25.0 | 3862673 | 1.57%
   17 | 1.2.1 | 2952346 | 1.20%
   18 | 1.0.1 | 2690006 | 1.09%
   19 | 0.22.0 | 2680710 | 1.09%
   20 | 1.2.0 | 2645339 | 1.07%
   21 | 0.24.1 | 2635411 | 1.07%
   
   - There are more than 60+% users downloaded the 1.x version in last 3 months
   - There are 26+% users downloaded the 0.23.2~1.0
   	
   
   
   [1] https://packaging.python.org/guides/analyzing-pypi-package-downloads/


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] viirya commented on a change in pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5

Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #34717:
URL: https://github.com/apache/spark/pull/34717#discussion_r758572738



##########
File path: python/docs/source/user_guide/sql/arrow_pandas.rst
##########
@@ -387,7 +387,7 @@ working with timestamps in ``pandas_udf``\s to get the best performance, see
 Recommended Pandas and PyArrow Versions
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-For usage with pyspark.sql, the minimum supported versions of Pandas is 0.23.2 and PyArrow is 1.0.0.
+For usage with pyspark.sql, the minimum supported versions of Pandas is 1.0.5 and PyArrow is 1.0.0.

Review comment:
       Should we mention there are some issues with versions like 1.0.0, 1.0.1?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Yikun commented on a change in pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5

Posted by GitBox <gi...@apache.org>.
Yikun commented on a change in pull request #34717:
URL: https://github.com/apache/spark/pull/34717#discussion_r758902796



##########
File path: python/pyspark/pandas/tests/test_series.py
##########
@@ -2209,12 +2209,12 @@ def test_mad(self):
         pser = pd.Series([1, 2, 3, 4], name="Koalas")
         psser = ps.from_pandas(pser)
 
-        self.assert_eq(pser.mad(), psser.mad())
+        self.assert_eq(pser.mad(), psser.mad(), almost=True)

Review comment:
       Addressed.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.0

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-980488878


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50141/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Yikun commented on a change in pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5

Posted by GitBox <gi...@apache.org>.
Yikun commented on a change in pull request #34717:
URL: https://github.com/apache/spark/pull/34717#discussion_r758898358



##########
File path: python/docs/source/user_guide/sql/arrow_pandas.rst
##########
@@ -387,7 +387,7 @@ working with timestamps in ``pandas_udf``\s to get the best performance, see
 Recommended Pandas and PyArrow Versions
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-For usage with pyspark.sql, the minimum supported versions of Pandas is 0.23.2 and PyArrow is 1.0.0.
+For usage with pyspark.sql, the minimum supported versions of Pandas is 1.0.5 and PyArrow is 1.0.0.

Review comment:
       How about?
   
   For usage with pyspark.sql, the minimum supported versions of Pandas is 1.0.5 and PyArrow is 1.0.0. Lower versions (such as there are some known issues under with v1.0.0, v1.0.1, see more in [link](https://github.com/apache/spark/pull/34717)) or higher versions may be used, however, compatibility and data correctness can not be guaranteed and should be verified by the user.
   
   Maybe need more suggestion from native speaker. T_T, and if it's necessary we could do it in next commits in this PR or followup.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Yikun commented on a change in pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5

Posted by GitBox <gi...@apache.org>.
Yikun commented on a change in pull request #34717:
URL: https://github.com/apache/spark/pull/34717#discussion_r758894595



##########
File path: python/pyspark/pandas/tests/test_series.py
##########
@@ -2209,12 +2209,12 @@ def test_mad(self):
         pser = pd.Series([1, 2, 3, 4], name="Koalas")
         psser = ps.from_pandas(pser)
 
-        self.assert_eq(pser.mad(), psser.mad())
+        self.assert_eq(pser.mad(), psser.mad(), almost=True)

Review comment:
       Yep, I tried `self.assert_eq(pser.mad(), psser.mad(), check_exact=False)`, it failed again. Because the test.mad() result value type is a numpy.float64 scalar, it would be matched to:
   
   https://github.com/apache/spark/blob/c38c617bcd04429e50310e4c7e0bd989c6555836/python/pyspark/testing/pandasutils.py#L237-L240
   
   So, only `almost=True` work.
   
   And I know your concern to make sure the type is right in here, but not only to check value. I notice that only last check is failed in my env, so we can keep top 3 check as original, and just make last check as almost=True, and also add a note in here.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Yikun commented on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.0

Posted by GitBox <gi...@apache.org>.
Yikun commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-981463864


   Complete all `pyspark-pandas-slow` test with:
   ```
   python/run-tests --modules=pyspark-pandas --parallelism=2 --python-executable=python3
   ```
   
   Serveral test cases failed in 1.0.1 due to `AttributeError: type object 'object' has no attribute 'dtype'` and **passed with pandas v1.0.5**.
   
   
   <details>
     <summary>Test failure details</summary>
   
   ```
   ======================================================================
   ERROR: test_astype (pyspark.pandas.tests.data_type_ops.test_categorical_ops.CategoricalOpsTest)
   ----------------------------------------------------------------------
   Traceback (most recent call last):
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/tests/data_type_ops/test_categorical_ops.py", line 204, in test_astype
       self.assert_eq(pser.astype(int), psser.astype(int))
     File "/Users/jiangyikun/spark/spark/python/pyspark/testing/pandasutils.py", line 224, in assert_eq
       robj = self._to_pandas(right)
     File "/Users/jiangyikun/spark/spark/python/pyspark/testing/pandasutils.py", line 245, in _to_pandas
       return obj.to_pandas()
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/series.py", line 1588, in to_pandas
       return self._to_pandas()
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/series.py", line 1594, in _to_pandas
       return self._to_internal_pandas().copy()
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/series.py", line 6349, in _to_internal_pandas
       return self._psdf._internal.to_pandas_frame[self.name]
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/utils.py", line 584, in wrapped_lazy_property
       setattr(self, attr_name, fn(self))
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/internal.py", line 1049, in to_pandas_frame
       pdf = sdf.toPandas()
     File "/Users/jiangyikun/spark/spark/python/pyspark/sql/pandas/conversion.py", line 185, in toPandas
       pdf = pd.DataFrame(columns=tmp_column_names).astype(
     File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/frame.py", line 435, in __init__
       mgr = init_dict(data, index, columns, dtype=dtype)
     File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/internals/construction.py", line 239, in init_dict
       val = construct_1d_arraylike_from_scalar(np.nan, len(index), nan_dtype)
     File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/dtypes/cast.py", line 1449, in construct_1d_arraylike_from_scalar
       dtype = dtype.dtype
   AttributeError: type object 'object' has no attribute 'dtype'
   
   ======================================================================
   ERROR: test_read_csv (pyspark.pandas.tests.test_csv.CsvTest)
   ----------------------------------------------------------------------
   Traceback (most recent call last):
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/tests/test_csv.py", line 151, in test_read_csv
       check(usecols=[])
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/tests/test_csv.py", line 138, in check
       self.assert_eq(expected, actual, almost=True)
     File "/Users/jiangyikun/spark/spark/python/pyspark/testing/pandasutils.py", line 224, in assert_eq
       robj = self._to_pandas(right)
     File "/Users/jiangyikun/spark/spark/python/pyspark/testing/pandasutils.py", line 245, in _to_pandas
       return obj.to_pandas()
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/frame.py", line 4856, in to_pandas
       return self._to_pandas()
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/frame.py", line 4862, in _to_pandas
       return self._internal.to_pandas_frame.copy()
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/utils.py", line 584, in wrapped_lazy_property
       setattr(self, attr_name, fn(self))
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/internal.py", line 1049, in to_pandas_frame
       pdf = sdf.toPandas()
     File "/Users/jiangyikun/spark/spark/python/pyspark/sql/pandas/conversion.py", line 185, in toPandas
       pdf = pd.DataFrame(columns=tmp_column_names).astype(
     File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/frame.py", line 435, in __init__
       mgr = init_dict(data, index, columns, dtype=dtype)
     File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/internals/construction.py", line 239, in init_dict
       val = construct_1d_arraylike_from_scalar(np.nan, len(index), nan_dtype)
     File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/dtypes/cast.py", line 1449, in construct_1d_arraylike_from_scalar
       dtype = dtype.dtype
   AttributeError: type object 'object' has no attribute 'dtype'
   
   ======================================================================
   ERROR: test_kde_plot (pyspark.pandas.tests.plot.test_frame_plot_plotly.DataFramePlotPlotlyTest)
   ----------------------------------------------------------------------
   Traceback (most recent call last):
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/tests/plot/test_frame_plot_plotly.py", line 262, in test_kde_plot
       actual = psdf.plot.kde(bw_method=5, ind=3)
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/plot/core.py", line 946, in kde
       return self(kind="kde", bw_method=bw_method, ind=ind, **kwargs)
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/plot/core.py", line 498, in __call__
       return plot_backend.plot_pandas_on_spark(plot_data, kind=kind, **kwargs)
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/plot/plotly.py", line 44, in plot_pandas_on_spark
       return plot_kde(data, **kwargs)
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/plot/plotly.py", line 202, in plot_kde
       pd.DataFrame(
     File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/frame.py", line 435, in __init__
       mgr = init_dict(data, index, columns, dtype=dtype)
     File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/internals/construction.py", line 254, in init_dict
       return arrays_to_mgr(arrays, data_names, index, columns, dtype=dtype)
     File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/internals/construction.py", line 69, in arrays_to_mgr
       arrays = _homogenize(arrays, index, dtype)
     File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/internals/construction.py", line 322, in _homogenize
       val = sanitize_array(
     File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/construction.py", line 465, in sanitize_array
       subarr = construct_1d_arraylike_from_scalar(value, len(index), dtype)
     File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/dtypes/cast.py", line 1461, in construct_1d_arraylike_from_scalar
       subarr = np.empty(length, dtype=dtype)
   TypeError: Cannot interpret '<attribute 'dtype' of 'numpy.generic' objects>' as a data type
   
   ======================================================================
   ERROR: test_kde_plot (pyspark.pandas.tests.plot.test_series_plot_plotly.SeriesPlotPlotlyTest)
   ----------------------------------------------------------------------
   Traceback (most recent call last):
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/tests/plot/test_series_plot_plotly.py", line 231, in test_kde_plot
       actual = psdf.a.plot.kde(bw_method=5, ind=3)
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/plot/core.py", line 946, in kde
       return self(kind="kde", bw_method=bw_method, ind=ind, **kwargs)
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/plot/core.py", line 498, in __call__
       return plot_backend.plot_pandas_on_spark(plot_data, kind=kind, **kwargs)
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/plot/plotly.py", line 44, in plot_pandas_on_spark
       return plot_kde(data, **kwargs)
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/plot/plotly.py", line 202, in plot_kde
       pd.DataFrame(
     File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/frame.py", line 435, in __init__
       mgr = init_dict(data, index, columns, dtype=dtype)
     File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/internals/construction.py", line 254, in init_dict
       return arrays_to_mgr(arrays, data_names, index, columns, dtype=dtype)
     File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/internals/construction.py", line 69, in arrays_to_mgr
       arrays = _homogenize(arrays, index, dtype)
     File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/internals/construction.py", line 322, in _homogenize
       val = sanitize_array(
     File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/construction.py", line 465, in sanitize_array
       subarr = construct_1d_arraylike_from_scalar(value, len(index), dtype)
     File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/dtypes/cast.py", line 1461, in construct_1d_arraylike_from_scalar
       subarr = np.empty(length, dtype=dtype)
   TypeError: Cannot interpret '<attribute 'dtype' of 'numpy.generic' objects>' as a data type
   
   ```
   </details>
   
   At this time, I prefer to update to **1.0.5**, I'm going to run `pyspark-pandas-slow` now.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-982292481


   Merged to master.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Yikun edited a comment on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.0

Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-981220341


   Sure, thanks for your suggestion, I'd like to update. and I added a simple test to install pandas v1.0.1 ~and run test on https://github.com/apache/spark/pull/34730 , wait for the result.~
   
   : (, Update: pandas only publish ubuntu wheel after v1.2....we have to install many deps, otherwise it would be failed when using `pip install pandas==1.0.1`,so I just install in my local env (macos, x86, yes have the 1.0.1 wheel) and running pip install 'pandas==1.0.1' and `python/run-tests --modules=pyspark-pandas,pyspark-pandas-slow --parallelism=2 --python-executable=python3` to test it.
   
   and looks like there were some testcase are failed:
   ```
   ======================================================================
   ERROR: test_astype (pyspark.pandas.tests.data_type_ops.test_categorical_ops.CategoricalOpsTest)
   ----------------------------------------------------------------------
   Traceback (most recent call last):
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/tests/data_type_ops/test_categorical_ops.py", line 204, in test_astype
       self.assert_eq(pser.astype(int), psser.astype(int))
     File "/Users/jiangyikun/spark/spark/python/pyspark/testing/pandasutils.py", line 224, in assert_eq
       robj = self._to_pandas(right)
     File "/Users/jiangyikun/spark/spark/python/pyspark/testing/pandasutils.py", line 245, in _to_pandas
       return obj.to_pandas()
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/series.py", line 1588, in to_pandas
       return self._to_pandas()
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/series.py", line 1594, in _to_pandas
       return self._to_internal_pandas().copy()
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/series.py", line 6349, in _to_internal_pandas
       return self._psdf._internal.to_pandas_frame[self.name]
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/utils.py", line 584, in wrapped_lazy_property
       setattr(self, attr_name, fn(self))
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/internal.py", line 1049, in to_pandas_frame
       pdf = sdf.toPandas()
     File "/Users/jiangyikun/spark/spark/python/pyspark/sql/pandas/conversion.py", line 185, in toPandas
       pdf = pd.DataFrame(columns=tmp_column_names).astype(
     File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/frame.py", line 435, in __init__
       mgr = init_dict(data, index, columns, dtype=dtype)
     File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/internals/construction.py", line 239, in init_dict
       val = construct_1d_arraylike_from_scalar(np.nan, len(index), nan_dtype)
     File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/dtypes/cast.py", line 1449, in construct_1d_arraylike_from_scalar
       dtype = dtype.dtype
   AttributeError: type object 'object' has no attribute 'dtype'
   
   ----------------------------------------------------------------------
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] rshkv edited a comment on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.0

Posted by GitBox <gi...@apache.org>.
rshkv edited a comment on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-981126049


   I noticed that `IntegralExtensionOpsTest.test_invert` fails on Pandas 1.0.0 and succeeds on 1.0.1 (error below). So maybe safer to recommend that version. Otherwise everything seems to work with 1.0.0.
   
   <details>
   <summary>Test failure</summary>
   
   ```
   ======================================================================
   ERROR [0.404s]: test_invert (pyspark.pandas.tests.data_type_ops.test_num_ops.IntegralExtensionOpsTest)
   ----------------------------------------------------------------------
   Traceback (most recent call last):
     File "/home/circleci/project/python/pyspark/testing/pandasutils.py", line 122, in assertPandasEqual
       **kwargs
     File "/opt/pyenv/versions/3.7.3/lib/python3.7/site-packages/pandas/_testing.py", line 1137, in assert_series_equal
       assert_attr_equal("dtype", left, right, obj=f"Attributes of {obj}")
     File "/opt/pyenv/versions/3.7.3/lib/python3.7/site-packages/pandas/_testing.py", line 772, in assert_attr_equal
       raise_assert_detail(obj, msg, left_attr, right_attr)
     File "/opt/pyenv/versions/3.7.3/lib/python3.7/site-packages/pandas/_testing.py", line 915, in raise_assert_detail
       raise AssertionError(msg)
   AssertionError: Attributes of Series are different
   
   Attribute "dtype" are different
   [left]:  object
   [right]: Int8
   
   The above exception was the direct cause of the following exception:
   
   Traceback (most recent call last):
     File "/home/circleci/project/python/pyspark/pandas/tests/data_type_ops/test_num_ops.py", line 498, in test_invert
       self.check_extension(~pser, ~psser)
     File "/home/circleci/project/python/pyspark/pandas/tests/data_type_ops/testing_utils.py", line 248, in check_extension
       self.assert_eq(left, right)
     File "/home/circleci/project/python/pyspark/testing/pandasutils.py", line 223, in assert_eq
       self.assertPandasEqual(lobj, robj, check_exact=check_exact)
     File "/home/circleci/project/python/pyspark/testing/pandasutils.py", line 130, in assertPandasEqual
       raise AssertionError(msg) from e
   AssertionError: Attributes of Series are different
   
   Attribute "dtype" are different
   [left]:  object
   [right]: Int8
   
   Left:
   dtype: object
   object
   
   Right:
   dtype: Int8
   Int8
   ```
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] ueshin commented on a change in pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5

Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #34717:
URL: https://github.com/apache/spark/pull/34717#discussion_r759540086



##########
File path: python/pyspark/pandas/tests/test_series.py
##########
@@ -2209,12 +2209,12 @@ def test_mad(self):
         pser = pd.Series([1, 2, 3, 4], name="Koalas")
         psser = ps.from_pandas(pser)
 
-        self.assert_eq(pser.mad(), psser.mad())
+        self.assert_eq(pser.mad(), psser.mad(), almost=True)

Review comment:
       Ah, got it. The result was not `DataFrame` or `Series`, then `almost=True` should be fine. Thanks.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Yikun commented on a change in pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5

Posted by GitBox <gi...@apache.org>.
Yikun commented on a change in pull request #34717:
URL: https://github.com/apache/spark/pull/34717#discussion_r758894595



##########
File path: python/pyspark/pandas/tests/test_series.py
##########
@@ -2209,12 +2209,12 @@ def test_mad(self):
         pser = pd.Series([1, 2, 3, 4], name="Koalas")
         psser = ps.from_pandas(pser)
 
-        self.assert_eq(pser.mad(), psser.mad())
+        self.assert_eq(pser.mad(), psser.mad(), almost=True)

Review comment:
       Yep, I tried `self.assert_eq(pser.mad(), psser.mad(), check_exact=False)`, it failed again. Because the test.mad() result value type is a numpy.float64 scalar, it would be matched to:
   
   https://github.com/apache/spark/blob/c38c617bcd04429e50310e4c7e0bd989c6555836/python/pyspark/testing/pandasutils.py#L237-L240
   
   So, only `almost=True` work.
   
   And I know your concern it's make sure the type is right in here, but not only to check value. I notice that only last check is failed in my env, so we can keep top 3 check as original, and just make last check as almost=True, and also add a note in here.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Yikun edited a comment on pull request #34717: Bump minimum pandas version to 1.0.0

Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-979659317


   Just to start the discussion, by using below sql according [1], we can get the all download stat of Pandas in last 3 months.
   ```SQL
   SELECT
     file.version AS file_version,
     COUNT(*) AS num_downloads,
   FROM `the-psf.pypi.file_downloads`
   WHERE file.project = 'pandas'
   AND 
     -- Only query the last 3 months of history
     DATE(timestamp)
       BETWEEN DATE_SUB(CURRENT_DATE(), INTERVAL 3 MONTH)
       AND CURRENT_DATE()
   GROUP BY `file_version`
   ORDER BY `num_downloads` DESC
   ```
   
   Here is the Top 20 data, about 77% of the overall data, complete result can be found in [here](https://gist.github.com/Yikun/a1d8168c316966db520cf8f8a43ff0bf):
     | version | number | percent
   -- | -- | -- | --
   1 | 0.25.3 | 35149221 | 14.28%
   2 | 1.1.5 | 28722806 | 11.67%
   3 | 1.3.4 | 20944236 | 8.51%
   4 | 1.3.3 | 16861573 | 6.85%
   5 | 0.24.2 | 13235233 | 5.38%
   6 | 1.0.5 | 9201989 | 3.74%
   7 | 1.3.2 | 9077326 | 3.69%
   8 | 1.2.5 | 7902532 | 3.21%
   9 | 1.2.4 | 5754284 | 2.34%
   10 | 1.1.4 | 5710439 | 2.32%
   11 | 1.1.0 | 4760847 | 1.93%
   12 | 1.1.2 | 4621441 | 1.88%
   13 | 1.2.3 | 4607043 | 1.87%
   14 | 1.0.3 | 4601230 | 1.87%
   15 | 0.23.4 | 4251044 | 1.73%
   16 | 0.25.0 | 3862673 | 1.57%
   17 | 1.2.1 | 2952346 | 1.20%
   18 | 1.0.1 | 2690006 | 1.09%
   19 | 0.22.0 | 2680710 | 1.09%
   20 | 1.2.0 | 2645339 | 1.07%
   21 | 0.24.1 | 2635411 | 1.07%
   
   - There are more than 60+% users downloaded the 1.x version in last 3 months
   - There are 26+% users downloaded version v0.23.2 to v1.0
   	
   
   
   [1] https://packaging.python.org/guides/analyzing-pypi-package-downloads/


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.0

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-980486828


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145671/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.0

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-980486828


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145671/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-981583510


   **[Test build #145720 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145720/testReport)** for PR 34717 at commit [`7b1de6d`](https://github.com/apache/spark/commit/7b1de6dba33a9313efe4eda2715ada024bfa9d0d).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] itholic commented on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5

Posted by GitBox <gi...@apache.org>.
itholic commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-982192818


   LGTM if remaining comments are resolved.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-982271491


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145744/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] ueshin commented on a change in pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5

Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #34717:
URL: https://github.com/apache/spark/pull/34717#discussion_r758685142



##########
File path: python/pyspark/pandas/tests/test_series.py
##########
@@ -2209,12 +2209,12 @@ def test_mad(self):
         pser = pd.Series([1, 2, 3, 4], name="Koalas")
         psser = ps.from_pandas(pser)
 
-        self.assert_eq(pser.mad(), psser.mad())
+        self.assert_eq(pser.mad(), psser.mad(), almost=True)

Review comment:
       Could you try with `check_exact=False` instead of `almost=True`?
   If it works, it's better because it still checks the internal structure of the Series / DataFrame, whereas `almost=True` checks only values.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] rshkv commented on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.0

Posted by GitBox <gi...@apache.org>.
rshkv commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-981126049


   I noticed that `IntegralExtensionOpsTest.test_invert` fails on Pandas 1.0.0 and succeeds on 1.0.1 (error below). So Maybe safer to recommend that version. Otherwise everything seems to work with 1.0.0.
   
   <details>
   <summary>Test failure</summary>
   
   ```
   ======================================================================
   ERROR [0.404s]: test_invert (pyspark.pandas.tests.data_type_ops.test_num_ops.IntegralExtensionOpsTest)
   ----------------------------------------------------------------------
   Traceback (most recent call last):
     File "/home/circleci/project/python/pyspark/testing/pandasutils.py", line 122, in assertPandasEqual
       **kwargs
     File "/opt/pyenv/versions/3.7.3/lib/python3.7/site-packages/pandas/_testing.py", line 1137, in assert_series_equal
       assert_attr_equal("dtype", left, right, obj=f"Attributes of {obj}")
     File "/opt/pyenv/versions/3.7.3/lib/python3.7/site-packages/pandas/_testing.py", line 772, in assert_attr_equal
       raise_assert_detail(obj, msg, left_attr, right_attr)
     File "/opt/pyenv/versions/3.7.3/lib/python3.7/site-packages/pandas/_testing.py", line 915, in raise_assert_detail
       raise AssertionError(msg)
   AssertionError: Attributes of Series are different
   
   Attribute "dtype" are different
   [left]:  object
   [right]: Int8
   
   The above exception was the direct cause of the following exception:
   
   Traceback (most recent call last):
     File "/home/circleci/project/python/pyspark/pandas/tests/data_type_ops/test_num_ops.py", line 498, in test_invert
       self.check_extension(~pser, ~psser)
     File "/home/circleci/project/python/pyspark/pandas/tests/data_type_ops/testing_utils.py", line 248, in check_extension
       self.assert_eq(left, right)
     File "/home/circleci/project/python/pyspark/testing/pandasutils.py", line 223, in assert_eq
       self.assertPandasEqual(lobj, robj, check_exact=check_exact)
     File "/home/circleci/project/python/pyspark/testing/pandasutils.py", line 130, in assertPandasEqual
       raise AssertionError(msg) from e
   AssertionError: Attributes of Series are different
   
   Attribute "dtype" are different
   [left]:  object
   [right]: Int8
   
   Left:
   dtype: object
   object
   
   Right:
   dtype: Int8
   Int8
   ```
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] srowen commented on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.0

Posted by GitBox <gi...@apache.org>.
srowen commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-981182681


   Yeah just require 1.0.1 for this reason


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #34717: Bump minimum pandas version to 1.0.0

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-979658449


   **[Test build #145645 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145645/testReport)** for PR 34717 at commit [`7986d55`](https://github.com/apache/spark/commit/7986d5591021a64ea454937e94654beab90a3042).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.0

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-980480329


   **[Test build #145671 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145671/testReport)** for PR 34717 at commit [`e521b76`](https://github.com/apache/spark/commit/e521b762e0d8d078e3505480d81e01e71d21024f).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Yikun edited a comment on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5

Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-981564669


   As a conclusion in here:
   - before v1.0.1: test failed due to https://github.com/apache/spark/pull/34717#issuecomment-981182023 https://github.com/apache/spark/pull/34717#issuecomment-981182023
   - before v1.0.5: test failed due to https://github.com/apache/spark/pull/34717#issuecomment-981463864
   - There are only a precision problem on v1.0.5, so just add `almost=True` to `assert_eq`, we can fix the testcase
   
   So, I bump minimum pandas version to v1.0.5, the v1.0.5 is also the latest version of [Pandas verions 1.0](https://pandas.pydata.org/docs/whatsnew/index.html#version-1-0).
   
   Ready for review. : )


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Yikun edited a comment on pull request #34717: Bump minimum pandas version to 1.0.0

Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-979659317


   Just to start the discussion, by using below sql according [1], we can got the all download stat of Pandas in last 3 months.
   ```SQL
   SELECT
     file.version AS file_version,
     COUNT(*) AS num_downloads,
   FROM `the-psf.pypi.file_downloads`
   WHERE file.project = 'pandas'
   AND 
     -- Only query the last 3 months of history
     DATE(timestamp)
       BETWEEN DATE_SUB(CURRENT_DATE(), INTERVAL 3 MONTH)
       AND CURRENT_DATE()
   GROUP BY `file_version`
   ORDER BY `num_downloads` DESC
   ```
   
   Here is the Top 20 data, about 77% of the overall data, complete result can be found in [here](https://gist.github.com/Yikun/a1d8168c316966db520cf8f8a43ff0bf):
     | version | number | percent
   -- | -- | -- | --
   1 | 0.25.3 | 35149221 | 14.28%
   2 | 1.1.5 | 28722806 | 11.67%
   3 | 1.3.4 | 20944236 | 8.51%
   4 | 1.3.3 | 16861573 | 6.85%
   5 | 0.24.2 | 13235233 | 5.38%
   6 | 1.0.5 | 9201989 | 3.74%
   7 | 1.3.2 | 9077326 | 3.69%
   8 | 1.2.5 | 7902532 | 3.21%
   9 | 1.2.4 | 5754284 | 2.34%
   10 | 1.1.4 | 5710439 | 2.32%
   11 | 1.1.0 | 4760847 | 1.93%
   12 | 1.1.2 | 4621441 | 1.88%
   13 | 1.2.3 | 4607043 | 1.87%
   14 | 1.0.3 | 4601230 | 1.87%
   15 | 0.23.4 | 4251044 | 1.73%
   16 | 0.25.0 | 3862673 | 1.57%
   17 | 1.2.1 | 2952346 | 1.20%
   18 | 1.0.1 | 2690006 | 1.09%
   19 | 0.22.0 | 2680710 | 1.09%
   20 | 1.2.0 | 2645339 | 1.07%
   21 | 0.24.1 | 2635411 | 1.07%
   
   - There are more than 60+% users downloaded the 1.x version in last 3 months
   - There are 26+% users downloaded version v0.23.2 to v1.0
   	
   
   
   [1] https://packaging.python.org/guides/analyzing-pypi-package-downloads/


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34717: Bump minimum pandas version to 1.0.0

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-979658449


   **[Test build #145645 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145645/testReport)** for PR 34717 at commit [`7986d55`](https://github.com/apache/spark/commit/7986d5591021a64ea454937e94654beab90a3042).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Yikun edited a comment on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.0

Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-981220341


   Sure, thanks for your suggestion, I'd like to update. and I added a simple test to install pandas v1.0.1 ~and run test on https://github.com/apache/spark/pull/34730 , wait for the result.~
   
   : (, Update: pandas only publish ubuntu wheel after v1.2....we have to install many deps, otherwise it would be failed when using `pip install pandas==1.0.1`,so I just install in my local env (macos, x86) and running pip install 'pandas==1.0.1' and `python/run-tests --modules=pyspark-pandas,pyspark-pandas-slow --parallelism=2 --python-executable=python3` to test it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-982248418


   **[Test build #145744 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145744/testReport)** for PR 34717 at commit [`054905f`](https://github.com/apache/spark/commit/054905f5074b8c543d2830e8e0500227f508a85f).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-982271491


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145744/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-982295989


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50215/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-982248418


   **[Test build #145744 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145744/testReport)** for PR 34717 at commit [`054905f`](https://github.com/apache/spark/commit/054905f5074b8c543d2830e8e0500227f508a85f).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Yikun edited a comment on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.0

Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-981463864


   Complete all `pyspark-pandas` test with:
   ```
   python/run-tests --modules=pyspark-pandas --parallelism=2 --python-executable=python3
   ```
   
   Serveral test cases failed (4 cases failed due to same issue) in 1.0.1 due to `AttributeError: type object 'object' has no attribute 'dtype'` and **passed with pandas v1.0.5** (It might be fixed in https://github.com/pandas-dev/pandas/pull/34667/).
   
   
   <details>
     <summary>Test failure details</summary>
   
   ```
   ======================================================================
   ERROR: test_astype (pyspark.pandas.tests.data_type_ops.test_categorical_ops.CategoricalOpsTest)
   ----------------------------------------------------------------------
   Traceback (most recent call last):
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/tests/data_type_ops/test_categorical_ops.py", line 204, in test_astype
       self.assert_eq(pser.astype(int), psser.astype(int))
     File "/Users/jiangyikun/spark/spark/python/pyspark/testing/pandasutils.py", line 224, in assert_eq
       robj = self._to_pandas(right)
     File "/Users/jiangyikun/spark/spark/python/pyspark/testing/pandasutils.py", line 245, in _to_pandas
       return obj.to_pandas()
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/series.py", line 1588, in to_pandas
       return self._to_pandas()
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/series.py", line 1594, in _to_pandas
       return self._to_internal_pandas().copy()
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/series.py", line 6349, in _to_internal_pandas
       return self._psdf._internal.to_pandas_frame[self.name]
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/utils.py", line 584, in wrapped_lazy_property
       setattr(self, attr_name, fn(self))
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/internal.py", line 1049, in to_pandas_frame
       pdf = sdf.toPandas()
     File "/Users/jiangyikun/spark/spark/python/pyspark/sql/pandas/conversion.py", line 185, in toPandas
       pdf = pd.DataFrame(columns=tmp_column_names).astype(
     File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/frame.py", line 435, in __init__
       mgr = init_dict(data, index, columns, dtype=dtype)
     File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/internals/construction.py", line 239, in init_dict
       val = construct_1d_arraylike_from_scalar(np.nan, len(index), nan_dtype)
     File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/dtypes/cast.py", line 1449, in construct_1d_arraylike_from_scalar
       dtype = dtype.dtype
   AttributeError: type object 'object' has no attribute 'dtype'
   
   ======================================================================
   ERROR: test_read_csv (pyspark.pandas.tests.test_csv.CsvTest)
   ----------------------------------------------------------------------
   Traceback (most recent call last):
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/tests/test_csv.py", line 151, in test_read_csv
       check(usecols=[])
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/tests/test_csv.py", line 138, in check
       self.assert_eq(expected, actual, almost=True)
     File "/Users/jiangyikun/spark/spark/python/pyspark/testing/pandasutils.py", line 224, in assert_eq
       robj = self._to_pandas(right)
     File "/Users/jiangyikun/spark/spark/python/pyspark/testing/pandasutils.py", line 245, in _to_pandas
       return obj.to_pandas()
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/frame.py", line 4856, in to_pandas
       return self._to_pandas()
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/frame.py", line 4862, in _to_pandas
       return self._internal.to_pandas_frame.copy()
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/utils.py", line 584, in wrapped_lazy_property
       setattr(self, attr_name, fn(self))
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/internal.py", line 1049, in to_pandas_frame
       pdf = sdf.toPandas()
     File "/Users/jiangyikun/spark/spark/python/pyspark/sql/pandas/conversion.py", line 185, in toPandas
       pdf = pd.DataFrame(columns=tmp_column_names).astype(
     File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/frame.py", line 435, in __init__
       mgr = init_dict(data, index, columns, dtype=dtype)
     File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/internals/construction.py", line 239, in init_dict
       val = construct_1d_arraylike_from_scalar(np.nan, len(index), nan_dtype)
     File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/dtypes/cast.py", line 1449, in construct_1d_arraylike_from_scalar
       dtype = dtype.dtype
   AttributeError: type object 'object' has no attribute 'dtype'
   
   ======================================================================
   ERROR: test_kde_plot (pyspark.pandas.tests.plot.test_frame_plot_plotly.DataFramePlotPlotlyTest)
   ----------------------------------------------------------------------
   Traceback (most recent call last):
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/tests/plot/test_frame_plot_plotly.py", line 262, in test_kde_plot
       actual = psdf.plot.kde(bw_method=5, ind=3)
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/plot/core.py", line 946, in kde
       return self(kind="kde", bw_method=bw_method, ind=ind, **kwargs)
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/plot/core.py", line 498, in __call__
       return plot_backend.plot_pandas_on_spark(plot_data, kind=kind, **kwargs)
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/plot/plotly.py", line 44, in plot_pandas_on_spark
       return plot_kde(data, **kwargs)
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/plot/plotly.py", line 202, in plot_kde
       pd.DataFrame(
     File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/frame.py", line 435, in __init__
       mgr = init_dict(data, index, columns, dtype=dtype)
     File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/internals/construction.py", line 254, in init_dict
       return arrays_to_mgr(arrays, data_names, index, columns, dtype=dtype)
     File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/internals/construction.py", line 69, in arrays_to_mgr
       arrays = _homogenize(arrays, index, dtype)
     File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/internals/construction.py", line 322, in _homogenize
       val = sanitize_array(
     File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/construction.py", line 465, in sanitize_array
       subarr = construct_1d_arraylike_from_scalar(value, len(index), dtype)
     File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/dtypes/cast.py", line 1461, in construct_1d_arraylike_from_scalar
       subarr = np.empty(length, dtype=dtype)
   TypeError: Cannot interpret '<attribute 'dtype' of 'numpy.generic' objects>' as a data type
   
   ======================================================================
   ERROR: test_kde_plot (pyspark.pandas.tests.plot.test_series_plot_plotly.SeriesPlotPlotlyTest)
   ----------------------------------------------------------------------
   Traceback (most recent call last):
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/tests/plot/test_series_plot_plotly.py", line 231, in test_kde_plot
       actual = psdf.a.plot.kde(bw_method=5, ind=3)
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/plot/core.py", line 946, in kde
       return self(kind="kde", bw_method=bw_method, ind=ind, **kwargs)
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/plot/core.py", line 498, in __call__
       return plot_backend.plot_pandas_on_spark(plot_data, kind=kind, **kwargs)
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/plot/plotly.py", line 44, in plot_pandas_on_spark
       return plot_kde(data, **kwargs)
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/plot/plotly.py", line 202, in plot_kde
       pd.DataFrame(
     File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/frame.py", line 435, in __init__
       mgr = init_dict(data, index, columns, dtype=dtype)
     File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/internals/construction.py", line 254, in init_dict
       return arrays_to_mgr(arrays, data_names, index, columns, dtype=dtype)
     File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/internals/construction.py", line 69, in arrays_to_mgr
       arrays = _homogenize(arrays, index, dtype)
     File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/internals/construction.py", line 322, in _homogenize
       val = sanitize_array(
     File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/construction.py", line 465, in sanitize_array
       subarr = construct_1d_arraylike_from_scalar(value, len(index), dtype)
     File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/dtypes/cast.py", line 1461, in construct_1d_arraylike_from_scalar
       subarr = np.empty(length, dtype=dtype)
   TypeError: Cannot interpret '<attribute 'dtype' of 'numpy.generic' objects>' as a data type
   
   ```
   </details>
   
   At this time, I prefer to update to **1.0.5**, I'm going to run `pyspark-pandas-slow` now.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.0

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-980483121


   **[Test build #145671 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145671/testReport)** for PR 34717 at commit [`e521b76`](https://github.com/apache/spark/commit/e521b762e0d8d078e3505480d81e01e71d21024f).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.0

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-980492640


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50141/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-981627078


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145720/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-982295989


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50215/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-981734229


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50189/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-982270701


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50215/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-981666882


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50189/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon closed pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5

Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #34717:
URL: https://github.com/apache/spark/pull/34717


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34717: Bump minimum pandas version to 1.0.0

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-979676293


   **[Test build #145645 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145645/testReport)** for PR 34717 at commit [`7986d55`](https://github.com/apache/spark/commit/7986d5591021a64ea454937e94654beab90a3042).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34717: Bump minimum pandas version to 1.0.0

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-979681958


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145645/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] xinrong-databricks commented on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.0

Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-981212899


   Thanks @Yikun, how do you think about bumping to `1.0.1`?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Yikun commented on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.0

Posted by GitBox <gi...@apache.org>.
Yikun commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-981220341


   Sure, thanks for your suggestion, I'd like to update. and I added a simple test to install pandas v1.0.1 and run test on https://github.com/apache/spark/pull/34730 , wait for the result.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-981583510


   **[Test build #145720 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145720/testReport)** for PR 34717 at commit [`7b1de6d`](https://github.com/apache/spark/commit/7b1de6dba33a9313efe4eda2715ada024bfa9d0d).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-981705646


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50189/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Yikun edited a comment on pull request #34717: Bump minimum pandas version to 1.0.0

Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-979659317


   Just to start the discussion:
   
   Here is top 20 download from pypi stat:
   
   By using below sql according [1], we can got the all download stat of Pandas in last 3 months.
   ```SQL
   SELECT
     file.version AS file_version,
     COUNT(*) AS num_downloads,
   FROM `the-psf.pypi.file_downloads`
   WHERE file.project = 'pandas'
   AND 
     -- Only query the last 3 months of history
     DATE(timestamp)
       BETWEEN DATE_SUB(CURRENT_DATE(), INTERVAL 3 MONTH)
       AND CURRENT_DATE()
   GROUP BY `file_version`
   ORDER BY `num_downloads` DESC
   ```
   
   Here is the Top 20 data, about 77% of the overall data:
     | version | number | percent
   -- | -- | -- | --
   1 | 0.25.3 | 35149221 | 14.28%
   2 | 1.1.5 | 28722806 | 11.67%
   3 | 1.3.4 | 20944236 | 8.51%
   4 | 1.3.3 | 16861573 | 6.85%
   5 | 0.24.2 | 13235233 | 5.38%
   6 | 1.0.5 | 9201989 | 3.74%
   7 | 1.3.2 | 9077326 | 3.69%
   8 | 1.2.5 | 7902532 | 3.21%
   9 | 1.2.4 | 5754284 | 2.34%
   10 | 1.1.4 | 5710439 | 2.32%
   11 | 1.1.0 | 4760847 | 1.93%
   12 | 1.1.2 | 4621441 | 1.88%
   13 | 1.2.3 | 4607043 | 1.87%
   14 | 1.0.3 | 4601230 | 1.87%
   15 | 0.23.4 | 4251044 | 1.73%
   16 | 0.25.0 | 3862673 | 1.57%
   17 | 1.2.1 | 2952346 | 1.20%
   18 | 1.0.1 | 2690006 | 1.09%
   19 | 0.22.0 | 2680710 | 1.09%
   20 | 1.2.0 | 2645339 | 1.07%
   21 | 0.24.1 | 2635411 | 1.07%
   
   [1] https://packaging.python.org/guides/analyzing-pypi-package-downloads/


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on pull request #34717: Bump minimum pandas version to 1.0.0

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-979660513


   Let's also update https://github.com/apache/spark/blob/master/python/docs/source/getting_started/install.rst#dependencies


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on pull request #34717: Bump minimum pandas version to 1.0.0

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-979660278


   cc @ueshin @xinrong-databricks @itholic  FYI


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34717: Bump minimum pandas version to 1.0.0

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-979699593


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50117/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-982260719


   **[Test build #145744 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145744/testReport)** for PR 34717 at commit [`054905f`](https://github.com/apache/spark/commit/054905f5074b8c543d2830e8e0500227f508a85f).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34717: Bump minimum pandas version to 1.0.0

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-979699593


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50117/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Yikun edited a comment on pull request #34717: Bump minimum pandas version to 1.0.0

Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-979659317


   Just to start the discussion, by using below sql according [1], we can got the all download stat of Pandas in last 3 months.
   ```SQL
   SELECT
     file.version AS file_version,
     COUNT(*) AS num_downloads,
   FROM `the-psf.pypi.file_downloads`
   WHERE file.project = 'pandas'
   AND 
     -- Only query the last 3 months of history
     DATE(timestamp)
       BETWEEN DATE_SUB(CURRENT_DATE(), INTERVAL 3 MONTH)
       AND CURRENT_DATE()
   GROUP BY `file_version`
   ORDER BY `num_downloads` DESC
   ```
   
   Here is the Top 20 data, about 77% of the overall data, complete result can be found in [here](https://gist.github.com/Yikun/a1d8168c316966db520cf8f8a43ff0bf):
     | version | number | percent
   -- | -- | -- | --
   1 | 0.25.3 | 35149221 | 14.28%
   2 | 1.1.5 | 28722806 | 11.67%
   3 | 1.3.4 | 20944236 | 8.51%
   4 | 1.3.3 | 16861573 | 6.85%
   5 | 0.24.2 | 13235233 | 5.38%
   6 | 1.0.5 | 9201989 | 3.74%
   7 | 1.3.2 | 9077326 | 3.69%
   8 | 1.2.5 | 7902532 | 3.21%
   9 | 1.2.4 | 5754284 | 2.34%
   10 | 1.1.4 | 5710439 | 2.32%
   11 | 1.1.0 | 4760847 | 1.93%
   12 | 1.1.2 | 4621441 | 1.88%
   13 | 1.2.3 | 4607043 | 1.87%
   14 | 1.0.3 | 4601230 | 1.87%
   15 | 0.23.4 | 4251044 | 1.73%
   16 | 0.25.0 | 3862673 | 1.57%
   17 | 1.2.1 | 2952346 | 1.20%
   18 | 1.0.1 | 2690006 | 1.09%
   19 | 0.22.0 | 2680710 | 1.09%
   20 | 1.2.0 | 2645339 | 1.07%
   21 | 0.24.1 | 2635411 | 1.07%
   
   [1] https://packaging.python.org/guides/analyzing-pypi-package-downloads/


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34717: Bump minimum pandas version to 1.0.0

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-979681958


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145645/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Yikun commented on pull request #34717: Bump minimum pandas version to 1.0.0

Posted by GitBox <gi...@apache.org>.
Yikun commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-979659317


   Here is top 20 download from pypi stat:
   
   By using below way according
   ```SQL
   SELECT
     file.version AS file_version,
     COUNT(*) AS num_downloads,
   FROM `the-psf.pypi.file_downloads`
   WHERE file.project = 'pandas'
   AND 
     -- Only query the last 3 months of history
     DATE(timestamp)
       BETWEEN DATE_SUB(CURRENT_DATE(), INTERVAL 3 MONTH)
       AND CURRENT_DATE()
   GROUP BY `file_version`
   ORDER BY `num_downloads` DESC
   ```
   
     | version | number | percent
   -- | -- | -- | --
   1 | 0.25.3 | 35149221 | 14.28%
   2 | 1.1.5 | 28722806 | 11.67%
   3 | 1.3.4 | 20944236 | 8.51%
   4 | 1.3.3 | 16861573 | 6.85%
   5 | 0.24.2 | 13235233 | 5.38%
   6 | 1.0.5 | 9201989 | 3.74%
   7 | 1.3.2 | 9077326 | 3.69%
   8 | 1.2.5 | 7902532 | 3.21%
   9 | 1.2.4 | 5754284 | 2.34%
   10 | 1.1.4 | 5710439 | 2.32%
   11 | 1.1.0 | 4760847 | 1.93%
   12 | 1.1.2 | 4621441 | 1.88%
   13 | 1.2.3 | 4607043 | 1.87%
   14 | 1.0.3 | 4601230 | 1.87%
   15 | 0.23.4 | 4251044 | 1.73%
   16 | 0.25.0 | 3862673 | 1.57%
   17 | 1.2.1 | 2952346 | 1.20%
   18 | 1.0.1 | 2690006 | 1.09%
   19 | 0.22.0 | 2680710 | 1.09%
   20 | 1.2.0 | 2645339 | 1.07%
   21 | 0.24.1 | 2635411 | 1.07%
   
   [1] https://packaging.python.org/guides/analyzing-pypi-package-downloads/


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-981734229


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50189/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.0

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-980492640


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50141/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Yikun commented on a change in pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5

Posted by GitBox <gi...@apache.org>.
Yikun commented on a change in pull request #34717:
URL: https://github.com/apache/spark/pull/34717#discussion_r758898358



##########
File path: python/docs/source/user_guide/sql/arrow_pandas.rst
##########
@@ -387,7 +387,7 @@ working with timestamps in ``pandas_udf``\s to get the best performance, see
 Recommended Pandas and PyArrow Versions
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-For usage with pyspark.sql, the minimum supported versions of Pandas is 0.23.2 and PyArrow is 1.0.0.
+For usage with pyspark.sql, the minimum supported versions of Pandas is 1.0.5 and PyArrow is 1.0.0.

Review comment:
       How about:
   
   For usage with pyspark.sql, the minimum supported versions of Pandas is **1.0.5** and PyArrow is 1.0.0. **Lower versions (such as there are some known issues under with v1.0.0, v1.0.1, see more in [link](https://github.com/apache/spark/pull/34717)) or** higher versions may be used, however, compatibility and data correctness can not be guaranteed and should be verified by the user.
   
   Maybe need more suggestion from native speaker. T_T, and if it's necessary we could do it in next commits in this PR or followup.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.0

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-980484750


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50141/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Yikun edited a comment on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.0

Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-981220341






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Yikun edited a comment on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5

Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-981564669


   As a conclusion in here:
   - before v1.0.1: test failed due to https://github.com/apache/spark/pull/34717#issuecomment-981182023 https://github.com/apache/spark/pull/34717#issuecomment-981182023
   - before v1.0.5: test failed due to https://github.com/apache/spark/pull/34717#issuecomment-981463864
   - v1.0.5: There are only a precision problem, so I just added `almost=True` to `assert_eq` to fix the testcase
   
   So, I bump minimum pandas version to v1.0.5, the v1.0.5 is also the latest version of [Pandas verions 1.0](https://pandas.pydata.org/docs/whatsnew/index.html#version-1-0).
   
   Ready for review. : )


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Yikun commented on a change in pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5

Posted by GitBox <gi...@apache.org>.
Yikun commented on a change in pull request #34717:
URL: https://github.com/apache/spark/pull/34717#discussion_r758894595



##########
File path: python/pyspark/pandas/tests/test_series.py
##########
@@ -2209,12 +2209,12 @@ def test_mad(self):
         pser = pd.Series([1, 2, 3, 4], name="Koalas")
         psser = ps.from_pandas(pser)
 
-        self.assert_eq(pser.mad(), psser.mad())
+        self.assert_eq(pser.mad(), psser.mad(), almost=True)

Review comment:
       Yep, I tried `self.assert_eq(pser.mad(), psser.mad(), check_exact=False)`, it failed again. Because the test.mad() result value type is a numpy.float64 scalar, it would be matched to:
   
   https://github.com/apache/spark/blob/c38c617bcd04429e50310e4c7e0bd989c6555836/python/pyspark/testing/pandasutils.py#L237-L240
   
   So, only `almost=True` work.
   
   And I know your concern it's make sure the type is right in here, but not only to check value. I notice that only last check is failed in my env, so we can keep top 3 check as original, and just make last check as almost=True.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.0

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-980480329


   **[Test build #145671 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145671/testReport)** for PR 34717 at commit [`e521b76`](https://github.com/apache/spark/commit/e521b762e0d8d078e3505480d81e01e71d21024f).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] itholic commented on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.0

Posted by GitBox <gi...@apache.org>.
itholic commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-981182023


   > I noticed that `IntegralExtensionOpsTest.test_invert` fails on Pandas 1.0.0 and succeeds on 1.0.1 (error below). So maybe safer to recommend that version. Otherwise everything seems to work with 1.0.0.
   
   Yeah, it seems the bug in pandas 1.0.0.
   
   - pandas 1.0.0
   ```python
   >>> pser = pd.Series([1, 2, 3, None], dtype="Int8")
   >>> pser
   0       1
   1       2
   2       3
   3    <NA>
   dtype: Int8
   >>> ~pser
   0      -2
   1      -3
   2      -4
   3    <NA>
   dtype: object  # this should've been `Int8`
   ```
   
   Resolved in pandas 1.0.1.
   
   - pandas 1.0.1
   ```python
   >>> pser = pd.Series([1, 2, 3, None], dtype="Int8")
   >>> pser
   0       1
   1       2
   2       3
   3    <NA>
   dtype: Int8
   >>> ~pser
   0      -2
   1      -3
   2      -4
   3    <NA>
   dtype: Int8
   ```
   
   For addressing this,
   
   1. manually separate this test for pandas 1.0.0.
   2. set the minimum pandas to version 1.0.1.
   
   Not sure which way is better, but I think we can just go with `2` if there is no reason to stick with the 1.0.0.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Yikun edited a comment on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.0

Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-981463864






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Yikun edited a comment on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.0

Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-981463864


   Complete all `pyspark-pandas` test with:
   ```
   python/run-tests --modules=pyspark-pandas --parallelism=2 --python-executable=python3
   ```
   
   Serveral test cases failed (5 failed due to same issue) in 1.0.1 due to `AttributeError: type object 'object' has no attribute 'dtype'` and **passed with pandas v1.0.5** (It might be fixed in https://github.com/pandas-dev/pandas/pull/34667/).
   
   
   <details>
     <summary>Test failure details</summary>
   
   ```
   ======================================================================
   ERROR: test_astype (pyspark.pandas.tests.data_type_ops.test_categorical_ops.CategoricalOpsTest)
   ----------------------------------------------------------------------
   Traceback (most recent call last):
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/tests/data_type_ops/test_categorical_ops.py", line 204, in test_astype
       self.assert_eq(pser.astype(int), psser.astype(int))
     File "/Users/jiangyikun/spark/spark/python/pyspark/testing/pandasutils.py", line 224, in assert_eq
       robj = self._to_pandas(right)
     File "/Users/jiangyikun/spark/spark/python/pyspark/testing/pandasutils.py", line 245, in _to_pandas
       return obj.to_pandas()
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/series.py", line 1588, in to_pandas
       return self._to_pandas()
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/series.py", line 1594, in _to_pandas
       return self._to_internal_pandas().copy()
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/series.py", line 6349, in _to_internal_pandas
       return self._psdf._internal.to_pandas_frame[self.name]
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/utils.py", line 584, in wrapped_lazy_property
       setattr(self, attr_name, fn(self))
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/internal.py", line 1049, in to_pandas_frame
       pdf = sdf.toPandas()
     File "/Users/jiangyikun/spark/spark/python/pyspark/sql/pandas/conversion.py", line 185, in toPandas
       pdf = pd.DataFrame(columns=tmp_column_names).astype(
     File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/frame.py", line 435, in __init__
       mgr = init_dict(data, index, columns, dtype=dtype)
     File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/internals/construction.py", line 239, in init_dict
       val = construct_1d_arraylike_from_scalar(np.nan, len(index), nan_dtype)
     File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/dtypes/cast.py", line 1449, in construct_1d_arraylike_from_scalar
       dtype = dtype.dtype
   AttributeError: type object 'object' has no attribute 'dtype'
   
   ======================================================================
   ERROR: test_read_csv (pyspark.pandas.tests.test_csv.CsvTest)
   ----------------------------------------------------------------------
   Traceback (most recent call last):
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/tests/test_csv.py", line 151, in test_read_csv
       check(usecols=[])
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/tests/test_csv.py", line 138, in check
       self.assert_eq(expected, actual, almost=True)
     File "/Users/jiangyikun/spark/spark/python/pyspark/testing/pandasutils.py", line 224, in assert_eq
       robj = self._to_pandas(right)
     File "/Users/jiangyikun/spark/spark/python/pyspark/testing/pandasutils.py", line 245, in _to_pandas
       return obj.to_pandas()
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/frame.py", line 4856, in to_pandas
       return self._to_pandas()
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/frame.py", line 4862, in _to_pandas
       return self._internal.to_pandas_frame.copy()
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/utils.py", line 584, in wrapped_lazy_property
       setattr(self, attr_name, fn(self))
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/internal.py", line 1049, in to_pandas_frame
       pdf = sdf.toPandas()
     File "/Users/jiangyikun/spark/spark/python/pyspark/sql/pandas/conversion.py", line 185, in toPandas
       pdf = pd.DataFrame(columns=tmp_column_names).astype(
     File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/frame.py", line 435, in __init__
       mgr = init_dict(data, index, columns, dtype=dtype)
     File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/internals/construction.py", line 239, in init_dict
       val = construct_1d_arraylike_from_scalar(np.nan, len(index), nan_dtype)
     File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/dtypes/cast.py", line 1449, in construct_1d_arraylike_from_scalar
       dtype = dtype.dtype
   AttributeError: type object 'object' has no attribute 'dtype'
   
   ======================================================================
   ERROR: test_kde_plot (pyspark.pandas.tests.plot.test_frame_plot_plotly.DataFramePlotPlotlyTest)
   ----------------------------------------------------------------------
   Traceback (most recent call last):
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/tests/plot/test_frame_plot_plotly.py", line 262, in test_kde_plot
       actual = psdf.plot.kde(bw_method=5, ind=3)
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/plot/core.py", line 946, in kde
       return self(kind="kde", bw_method=bw_method, ind=ind, **kwargs)
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/plot/core.py", line 498, in __call__
       return plot_backend.plot_pandas_on_spark(plot_data, kind=kind, **kwargs)
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/plot/plotly.py", line 44, in plot_pandas_on_spark
       return plot_kde(data, **kwargs)
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/plot/plotly.py", line 202, in plot_kde
       pd.DataFrame(
     File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/frame.py", line 435, in __init__
       mgr = init_dict(data, index, columns, dtype=dtype)
     File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/internals/construction.py", line 254, in init_dict
       return arrays_to_mgr(arrays, data_names, index, columns, dtype=dtype)
     File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/internals/construction.py", line 69, in arrays_to_mgr
       arrays = _homogenize(arrays, index, dtype)
     File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/internals/construction.py", line 322, in _homogenize
       val = sanitize_array(
     File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/construction.py", line 465, in sanitize_array
       subarr = construct_1d_arraylike_from_scalar(value, len(index), dtype)
     File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/dtypes/cast.py", line 1461, in construct_1d_arraylike_from_scalar
       subarr = np.empty(length, dtype=dtype)
   TypeError: Cannot interpret '<attribute 'dtype' of 'numpy.generic' objects>' as a data type
   
   ======================================================================
   ERROR: test_kde_plot (pyspark.pandas.tests.plot.test_series_plot_plotly.SeriesPlotPlotlyTest)
   ----------------------------------------------------------------------
   Traceback (most recent call last):
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/tests/plot/test_series_plot_plotly.py", line 231, in test_kde_plot
       actual = psdf.a.plot.kde(bw_method=5, ind=3)
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/plot/core.py", line 946, in kde
       return self(kind="kde", bw_method=bw_method, ind=ind, **kwargs)
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/plot/core.py", line 498, in __call__
       return plot_backend.plot_pandas_on_spark(plot_data, kind=kind, **kwargs)
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/plot/plotly.py", line 44, in plot_pandas_on_spark
       return plot_kde(data, **kwargs)
     File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/plot/plotly.py", line 202, in plot_kde
       pd.DataFrame(
     File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/frame.py", line 435, in __init__
       mgr = init_dict(data, index, columns, dtype=dtype)
     File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/internals/construction.py", line 254, in init_dict
       return arrays_to_mgr(arrays, data_names, index, columns, dtype=dtype)
     File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/internals/construction.py", line 69, in arrays_to_mgr
       arrays = _homogenize(arrays, index, dtype)
     File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/internals/construction.py", line 322, in _homogenize
       val = sanitize_array(
     File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/construction.py", line 465, in sanitize_array
       subarr = construct_1d_arraylike_from_scalar(value, len(index), dtype)
     File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/dtypes/cast.py", line 1461, in construct_1d_arraylike_from_scalar
       subarr = np.empty(length, dtype=dtype)
   TypeError: Cannot interpret '<attribute 'dtype' of 'numpy.generic' objects>' as a data type
   
   ```
   </details>
   
   At this time, I prefer to update to **1.0.5**, I'm going to run `pyspark-pandas-slow` now.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] ueshin commented on a change in pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5

Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #34717:
URL: https://github.com/apache/spark/pull/34717#discussion_r758685142



##########
File path: python/pyspark/pandas/tests/test_series.py
##########
@@ -2209,12 +2209,12 @@ def test_mad(self):
         pser = pd.Series([1, 2, 3, 4], name="Koalas")
         psser = ps.from_pandas(pser)
 
-        self.assert_eq(pser.mad(), psser.mad())
+        self.assert_eq(pser.mad(), psser.mad(), almost=True)

Review comment:
       Could you try with `check_exact=False` instead of `almost=True`?
   If it works, it's better because it still checks the internal structure of the Series / DataFrame; otherwise `almost=True` checks only values.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Yikun commented on a change in pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5

Posted by GitBox <gi...@apache.org>.
Yikun commented on a change in pull request #34717:
URL: https://github.com/apache/spark/pull/34717#discussion_r758895254



##########
File path: python/pyspark/pandas/tests/test_series.py
##########
@@ -2209,12 +2209,12 @@ def test_mad(self):
         pser = pd.Series([1, 2, 3, 4], name="Koalas")
         psser = ps.from_pandas(pser)
 
-        self.assert_eq(pser.mad(), psser.mad())
+        self.assert_eq(pser.mad(), psser.mad(), almost=True)

Review comment:
       And one more thing, this test case failed in my env always, even the latest pandas v1.3.4, so looks like a precision issue cross platfroms.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Yikun commented on a change in pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5

Posted by GitBox <gi...@apache.org>.
Yikun commented on a change in pull request #34717:
URL: https://github.com/apache/spark/pull/34717#discussion_r758898358



##########
File path: python/docs/source/user_guide/sql/arrow_pandas.rst
##########
@@ -387,7 +387,7 @@ working with timestamps in ``pandas_udf``\s to get the best performance, see
 Recommended Pandas and PyArrow Versions
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-For usage with pyspark.sql, the minimum supported versions of Pandas is 0.23.2 and PyArrow is 1.0.0.
+For usage with pyspark.sql, the minimum supported versions of Pandas is 1.0.5 and PyArrow is 1.0.0.

Review comment:
       How about?
   
   For usage with pyspark.sql, the minimum supported versions of Pandas is 1.0.5 and PyArrow is 1.0.0. Lower versions (such as there are some known issue under with v1.0.0, v1.0.1, see more in [link](https://github.com/apache/spark/pull/34717)) or higher versions may be used, however, compatibility and data correctness can not be guaranteed and should be verified by the user.
   
   Maybe need more suggestion from native speaker. T_T

##########
File path: python/docs/source/user_guide/sql/arrow_pandas.rst
##########
@@ -387,7 +387,7 @@ working with timestamps in ``pandas_udf``\s to get the best performance, see
 Recommended Pandas and PyArrow Versions
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-For usage with pyspark.sql, the minimum supported versions of Pandas is 0.23.2 and PyArrow is 1.0.0.
+For usage with pyspark.sql, the minimum supported versions of Pandas is 1.0.5 and PyArrow is 1.0.0.

Review comment:
       How about?
   
   For usage with pyspark.sql, the minimum supported versions of Pandas is 1.0.5 and PyArrow is 1.0.0. Lower versions (such as there are some known issues under with v1.0.0, v1.0.1, see more in [link](https://github.com/apache/spark/pull/34717)) or higher versions may be used, however, compatibility and data correctness can not be guaranteed and should be verified by the user.
   
   Maybe need more suggestion from native speaker. T_T




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Yikun edited a comment on pull request #34717: Bump minimum pandas version to 1.0.0

Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-979659317


   Just to start the discussion:
   
   Here is top 20 download from pypi stat:
   
   By using below sql according [1], we can got the top 20 download version of Pandas in last 3 months.
   ```SQL
   SELECT
     file.version AS file_version,
     COUNT(*) AS num_downloads,
   FROM `the-psf.pypi.file_downloads`
   WHERE file.project = 'pandas'
   AND 
     -- Only query the last 3 months of history
     DATE(timestamp)
       BETWEEN DATE_SUB(CURRENT_DATE(), INTERVAL 3 MONTH)
       AND CURRENT_DATE()
   GROUP BY `file_version`
   ORDER BY `num_downloads` DESC
   ```
   
     | version | number | percent
   -- | -- | -- | --
   1 | 0.25.3 | 35149221 | 14.28%
   2 | 1.1.5 | 28722806 | 11.67%
   3 | 1.3.4 | 20944236 | 8.51%
   4 | 1.3.3 | 16861573 | 6.85%
   5 | 0.24.2 | 13235233 | 5.38%
   6 | 1.0.5 | 9201989 | 3.74%
   7 | 1.3.2 | 9077326 | 3.69%
   8 | 1.2.5 | 7902532 | 3.21%
   9 | 1.2.4 | 5754284 | 2.34%
   10 | 1.1.4 | 5710439 | 2.32%
   11 | 1.1.0 | 4760847 | 1.93%
   12 | 1.1.2 | 4621441 | 1.88%
   13 | 1.2.3 | 4607043 | 1.87%
   14 | 1.0.3 | 4601230 | 1.87%
   15 | 0.23.4 | 4251044 | 1.73%
   16 | 0.25.0 | 3862673 | 1.57%
   17 | 1.2.1 | 2952346 | 1.20%
   18 | 1.0.1 | 2690006 | 1.09%
   19 | 0.22.0 | 2680710 | 1.09%
   20 | 1.2.0 | 2645339 | 1.07%
   21 | 0.24.1 | 2635411 | 1.07%
   
   [1] https://packaging.python.org/guides/analyzing-pypi-package-downloads/


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Yikun edited a comment on pull request #34717: Bump minimum pandas version to 1.0.0

Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-979659317


   Just to start the discussion:
   
   Here is top 20 download from pypi stat:
   
   By using below way according [1]
   ```SQL
   SELECT
     file.version AS file_version,
     COUNT(*) AS num_downloads,
   FROM `the-psf.pypi.file_downloads`
   WHERE file.project = 'pandas'
   AND 
     -- Only query the last 3 months of history
     DATE(timestamp)
       BETWEEN DATE_SUB(CURRENT_DATE(), INTERVAL 3 MONTH)
       AND CURRENT_DATE()
   GROUP BY `file_version`
   ORDER BY `num_downloads` DESC
   ```
   
     | version | number | percent
   -- | -- | -- | --
   1 | 0.25.3 | 35149221 | 14.28%
   2 | 1.1.5 | 28722806 | 11.67%
   3 | 1.3.4 | 20944236 | 8.51%
   4 | 1.3.3 | 16861573 | 6.85%
   5 | 0.24.2 | 13235233 | 5.38%
   6 | 1.0.5 | 9201989 | 3.74%
   7 | 1.3.2 | 9077326 | 3.69%
   8 | 1.2.5 | 7902532 | 3.21%
   9 | 1.2.4 | 5754284 | 2.34%
   10 | 1.1.4 | 5710439 | 2.32%
   11 | 1.1.0 | 4760847 | 1.93%
   12 | 1.1.2 | 4621441 | 1.88%
   13 | 1.2.3 | 4607043 | 1.87%
   14 | 1.0.3 | 4601230 | 1.87%
   15 | 0.23.4 | 4251044 | 1.73%
   16 | 0.25.0 | 3862673 | 1.57%
   17 | 1.2.1 | 2952346 | 1.20%
   18 | 1.0.1 | 2690006 | 1.09%
   19 | 0.22.0 | 2680710 | 1.09%
   20 | 1.2.0 | 2645339 | 1.07%
   21 | 0.24.1 | 2635411 | 1.07%
   
   [1] https://packaging.python.org/guides/analyzing-pypi-package-downloads/


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Yikun edited a comment on pull request #34717: Bump minimum pandas version to 1.0.0

Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-979659317


   Here is top 20 download from pypi stat:
   
   By using below way according [1]
   ```SQL
   SELECT
     file.version AS file_version,
     COUNT(*) AS num_downloads,
   FROM `the-psf.pypi.file_downloads`
   WHERE file.project = 'pandas'
   AND 
     -- Only query the last 3 months of history
     DATE(timestamp)
       BETWEEN DATE_SUB(CURRENT_DATE(), INTERVAL 3 MONTH)
       AND CURRENT_DATE()
   GROUP BY `file_version`
   ORDER BY `num_downloads` DESC
   ```
   
     | version | number | percent
   -- | -- | -- | --
   1 | 0.25.3 | 35149221 | 14.28%
   2 | 1.1.5 | 28722806 | 11.67%
   3 | 1.3.4 | 20944236 | 8.51%
   4 | 1.3.3 | 16861573 | 6.85%
   5 | 0.24.2 | 13235233 | 5.38%
   6 | 1.0.5 | 9201989 | 3.74%
   7 | 1.3.2 | 9077326 | 3.69%
   8 | 1.2.5 | 7902532 | 3.21%
   9 | 1.2.4 | 5754284 | 2.34%
   10 | 1.1.4 | 5710439 | 2.32%
   11 | 1.1.0 | 4760847 | 1.93%
   12 | 1.1.2 | 4621441 | 1.88%
   13 | 1.2.3 | 4607043 | 1.87%
   14 | 1.0.3 | 4601230 | 1.87%
   15 | 0.23.4 | 4251044 | 1.73%
   16 | 0.25.0 | 3862673 | 1.57%
   17 | 1.2.1 | 2952346 | 1.20%
   18 | 1.0.1 | 2690006 | 1.09%
   19 | 0.22.0 | 2680710 | 1.09%
   20 | 1.2.0 | 2645339 | 1.07%
   21 | 0.24.1 | 2635411 | 1.07%
   
   [1] https://packaging.python.org/guides/analyzing-pypi-package-downloads/


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34717: Bump minimum pandas version to 1.0.0

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-979696936


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50117/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-981608086


   **[Test build #145720 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145720/testReport)** for PR 34717 at commit [`7b1de6d`](https://github.com/apache/spark/commit/7b1de6dba33a9313efe4eda2715ada024bfa9d0d).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34717:
URL: https://github.com/apache/spark/pull/34717#issuecomment-981627078


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145720/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org