You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/06/24 07:19:52 UTC
[GitHub] [spark] itholic opened a new pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
itholic opened a new pull request #33054:
URL: https://github.com/apache/spark/pull/33054
### What changes were proposed in this pull request?
This PR proposes move `to_pandas_on_spark` function from `pyspark.pandas.frame` to `pyspark.sql.dataframe`, and added the related tests to the PySpark DataFrame tests.
### Why are the changes needed?
Because now the Koalas is ported into PySpark, so we don't need to Spark auto-patch anymore.
And also `to_pandas_on_spark` is belongs to the pandas-on-Spark DataFrame doesn't look make sense.
### Does this PR introduce _any_ user-facing change?
No, it's kinda internal refactoring stuff.
### How was this patch tested?
Added the related tests and manually check they're passed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33054:
URL: https://github.com/apache/spark/pull/33054#issuecomment-868281740
**[Test build #140308 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140308/testReport)** for PR 33054 at commit [`b92032f`](https://github.com/apache/spark/commit/b92032f4dc0b5d24ae1065fbb91a3ffadd24bf4d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33054:
URL: https://github.com/apache/spark/pull/33054#issuecomment-867421659
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140249/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33054:
URL: https://github.com/apache/spark/pull/33054#discussion_r658388558
##########
File path: python/pyspark/sql/tests/test_dataframe.py
##########
@@ -855,6 +855,22 @@ def test_df_show(self):
with self.assertRaisesRegex(TypeError, "Parameter 'truncate=foo'"):
df.show(truncate='foo')
+ @unittest.skipIf(
+ not have_pandas or not have_pyarrow,
+ pandas_requirement_message or pyarrow_requirement_message) # type: ignore
+ def test_to_pandas_on_spark(self):
+ from pyspark.pandas.frame import DataFrame
+ from pandas.testing import assert_frame_equal
+
+ sdf = self.spark.createDataFrame([("a", 1), ("b", 2), ("c", 3)], ["Col1", "Col2"])
+ psdf_from_sdf = sdf.to_pandas_on_spark()
+ psdf_from_sdf_with_index = sdf.to_pandas_on_spark(index_col="Col1")
+ psdf = DataFrame({"Col1": ["a", "b", "c"], "Col2": [1, 2, 3]})
Review comment:
maybe directly create a pandas DataFrame?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33054:
URL: https://github.com/apache/spark/pull/33054#issuecomment-868123787
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33054:
URL: https://github.com/apache/spark/pull/33054#issuecomment-867411977
**[Test build #140249 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140249/testReport)** for PR 33054 at commit [`32c24d7`](https://github.com/apache/spark/commit/32c24d7f8a7652dbc03457d9b1963fbc03354a88).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33054:
URL: https://github.com/apache/spark/pull/33054#discussion_r657770107
##########
File path: python/pyspark/sql/dataframe.py
##########
@@ -2695,6 +2695,62 @@ def writeTo(self, table):
"""
return DataFrameWriterV2(self, table)
+ def to_pandas_on_spark(self, index_col=None):
+ """
+ Converts the existing DataFrame into a pandas-on-Spark DataFrame.
+
+ If a pandas-on-Spark DataFrame is converted to a Spark DataFrame and then back
+ to pandas-on-Spark, it will lose the index information and the original index
+ will be turned into a normal column.
+
+ Parameters
+ ----------
+ index_col: str or list of str, optional, default: None
+ Index column of table in Spark.
+
+ See Also
+ --------
+ DataFrame.to_spark
Review comment:
`pyspark.pandas.DataFrame.to_spark`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #33054:
URL: https://github.com/apache/spark/pull/33054#issuecomment-868220295
Are you sure if this is the same issue? In our case, we're importing it without try-except:
https://github.com/apache/spark/blob/master/python/pyspark/pandas/plot/core.py#L23
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33054:
URL: https://github.com/apache/spark/pull/33054#issuecomment-868302420
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44840/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #33054:
URL: https://github.com/apache/spark/pull/33054#discussion_r658209332
##########
File path: python/pyspark/sql/dataframe.py
##########
@@ -2695,6 +2695,62 @@ def writeTo(self, table):
"""
return DataFrameWriterV2(self, table)
+ def to_pandas_on_spark(self, index_col=None):
Review comment:
We need to add the signature in `dataframe.pyi` file?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33054:
URL: https://github.com/apache/spark/pull/33054#issuecomment-868163557
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44825/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33054:
URL: https://github.com/apache/spark/pull/33054#issuecomment-867461504
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44778/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33054:
URL: https://github.com/apache/spark/pull/33054#issuecomment-867461467
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44778/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33054:
URL: https://github.com/apache/spark/pull/33054#issuecomment-868153282
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44823/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33054:
URL: https://github.com/apache/spark/pull/33054#issuecomment-868219290
**[Test build #140304 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140304/testReport)** for PR 33054 at commit [`7840c31`](https://github.com/apache/spark/commit/7840c31259f808fbdeb3599f0d8e267f0f96e0b2).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33054:
URL: https://github.com/apache/spark/pull/33054#issuecomment-868163557
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44825/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33054:
URL: https://github.com/apache/spark/pull/33054#issuecomment-868150025
**[Test build #140294 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140294/testReport)** for PR 33054 at commit [`0566b91`](https://github.com/apache/spark/commit/0566b91c37571297be6e2795145596aed5be274a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33054:
URL: https://github.com/apache/spark/pull/33054#discussion_r658387656
##########
File path: python/pyspark/sql/dataframe.pyi
##########
@@ -50,6 +50,8 @@ from pyspark.storagelevel import StorageLevel
from pyspark.sql.pandas.conversion import PandasConversionMixin
from pyspark.sql.pandas.map_ops import PandasMapOpsMixin
Review comment:
no space?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33054:
URL: https://github.com/apache/spark/pull/33054#discussion_r657769439
##########
File path: python/pyspark/sql/tests/test_dataframe.py
##########
@@ -855,6 +855,19 @@ def test_df_show(self):
with self.assertRaisesRegex(TypeError, "Parameter 'truncate=foo'"):
df.show(truncate='foo')
+ def test_to_pandas_on_spark(self):
Review comment:
You should run this conditionally. See `python/pyspark/sql/tests/test_arrow.py`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33054:
URL: https://github.com/apache/spark/pull/33054#issuecomment-868291727
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33054:
URL: https://github.com/apache/spark/pull/33054#discussion_r658387506
##########
File path: python/pyspark/sql/dataframe.pyi
##########
@@ -267,6 +269,7 @@ class DataFrame(PandasMapOpsMixin, PandasConversionMixin):
def semanticHash(self) -> int: ...
def inputFiles(self) -> List[str]: ...
def writeTo(self, table: str) -> DataFrameWriterV2: ...
+ def to_pandas_on_spark(self, index_col: None) -> PandasOnSparkDataFrame: ...
Review comment:
`Optional[Union[str, List[str]]] = None`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on a change in pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
itholic commented on a change in pull request #33054:
URL: https://github.com/apache/spark/pull/33054#discussion_r659426716
##########
File path: python/pyspark/pandas/plot/core.py
##########
@@ -20,7 +20,7 @@
import pandas as pd
import numpy as np
from pyspark.ml.feature import Bucketizer
-from pyspark.mllib.stat import KernelDensity
+from pyspark.mllib.stat.KernelDensity import KernelDensity
Review comment:
Otherwise it raises `mypy checks failed:
python/pyspark/pandas/plot/core.py:23: error: Name 'KernelDensity' already defined (by an import)` for some reason.
Anyway I think it's really weird because seems like my changes don't affect to the `KernelDensity` side at all, but the `mypy` check started to fail.
And the more weird fact is that the `KernelDensity` is only imported here across all project, but it suddenly says already defined.
Let me just ignore this case for now, and investigate more in mypy side. I couldn't find the related bug so far.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33054:
URL: https://github.com/apache/spark/pull/33054#issuecomment-868274094
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44835/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33054:
URL: https://github.com/apache/spark/pull/33054#issuecomment-868218106
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140302/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on a change in pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
itholic commented on a change in pull request #33054:
URL: https://github.com/apache/spark/pull/33054#discussion_r659426716
##########
File path: python/pyspark/pandas/plot/core.py
##########
@@ -20,7 +20,7 @@
import pandas as pd
import numpy as np
from pyspark.ml.feature import Bucketizer
-from pyspark.mllib.stat import KernelDensity
+from pyspark.mllib.stat.KernelDensity import KernelDensity
Review comment:
Otherwise it raises `mypy checks failed:
python/pyspark/pandas/plot/core.py:23: error: Name 'KernelDensity' already defined (by an import)` for some reason.
Anyway I think it's really weird because seems like my changes don't affect to the `KernelDensity` side at all, but the `mypy` check started to fail.
And the more weird fact is that the `KernelDensity` is only imported here across whole project (pandas-on-Spark), but it suddenly says already defined.
Let me just ignore this case for now, and investigate more in mypy side. I couldn't find the related bug so far.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #33054:
URL: https://github.com/apache/spark/pull/33054#issuecomment-869297320
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #33054:
URL: https://github.com/apache/spark/pull/33054#issuecomment-868150886
@itholic the linter failed. can you take a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33054:
URL: https://github.com/apache/spark/pull/33054#issuecomment-868326320
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33054:
URL: https://github.com/apache/spark/pull/33054#issuecomment-868326320
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33054:
URL: https://github.com/apache/spark/pull/33054#issuecomment-868157232
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33054:
URL: https://github.com/apache/spark/pull/33054#issuecomment-868265537
**[Test build #140308 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140308/testReport)** for PR 33054 at commit [`b92032f`](https://github.com/apache/spark/commit/b92032f4dc0b5d24ae1065fbb91a3ffadd24bf4d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33054:
URL: https://github.com/apache/spark/pull/33054#issuecomment-868123787
**[Test build #140292 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140292/testReport)** for PR 33054 at commit [`b162be4`](https://github.com/apache/spark/commit/b162be4cff45f2ced8c039dd63db46e1f9107dab).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33054:
URL: https://github.com/apache/spark/pull/33054#discussion_r657770758
##########
File path: python/pyspark/sql/dataframe.py
##########
@@ -2695,6 +2695,62 @@ def writeTo(self, table):
"""
return DataFrameWriterV2(self, table)
+ def to_pandas_on_spark(self, index_col=None):
+ """
+ Converts the existing DataFrame into a pandas-on-Spark DataFrame.
+
+ If a pandas-on-Spark DataFrame is converted to a Spark DataFrame and then back
+ to pandas-on-Spark, it will lose the index information and the original index
+ will be turned into a normal column.
+
+ Parameters
+ ----------
+ index_col: str or list of str, optional, default: None
+ Index column of table in Spark.
+
+ See Also
+ --------
+ DataFrame.to_spark
+
+ Examples
+ --------
+ >>> df = spark.createDataFrame([("a", 1), ("b", 2), ("c", 3)], ["Col1", "Col2"])
+
+ >>> psdf = df.to_pandas_on_spark()
Review comment:
I think you can skip those tests for now. See `PandasConversionMixin.toPandas` as an example.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33054:
URL: https://github.com/apache/spark/pull/33054#issuecomment-868251582
**[Test build #140304 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140304/testReport)** for PR 33054 at commit [`7840c31`](https://github.com/apache/spark/commit/7840c31259f808fbdeb3599f0d8e267f0f96e0b2).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33054:
URL: https://github.com/apache/spark/pull/33054#issuecomment-868218106
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140302/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33054:
URL: https://github.com/apache/spark/pull/33054#issuecomment-868267572
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44833/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33054:
URL: https://github.com/apache/spark/pull/33054#issuecomment-868211955
**[Test build #140302 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140302/testReport)** for PR 33054 at commit [`658f80d`](https://github.com/apache/spark/commit/658f80d45df7333c3ad0d7d137e02bce96735a1b).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33054:
URL: https://github.com/apache/spark/pull/33054#issuecomment-868267548
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44833/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
itholic commented on pull request #33054:
URL: https://github.com/apache/spark/pull/33054#issuecomment-868122046
Oh, just added. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33054:
URL: https://github.com/apache/spark/pull/33054#issuecomment-868157235
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33054:
URL: https://github.com/apache/spark/pull/33054#issuecomment-868290665
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44839/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33054:
URL: https://github.com/apache/spark/pull/33054#issuecomment-868156442
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44823/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33054:
URL: https://github.com/apache/spark/pull/33054#issuecomment-868219290
**[Test build #140304 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140304/testReport)** for PR 33054 at commit [`7840c31`](https://github.com/apache/spark/commit/7840c31259f808fbdeb3599f0d8e267f0f96e0b2).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33054:
URL: https://github.com/apache/spark/pull/33054#issuecomment-868262286
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140304/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33054:
URL: https://github.com/apache/spark/pull/33054#issuecomment-868158777
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44825/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33054:
URL: https://github.com/apache/spark/pull/33054#issuecomment-868202158
**[Test build #140302 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140302/testReport)** for PR 33054 at commit [`658f80d`](https://github.com/apache/spark/commit/658f80d45df7333c3ad0d7d137e02bce96735a1b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic edited a comment on pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
itholic edited a comment on pull request #33054:
URL: https://github.com/apache/spark/pull/33054#issuecomment-868201074
~~Seems like it's a known bug in mypy (https://github.com/python/mypy/issues/1153).~~
~~I'd ignore this case for now.~~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33054:
URL: https://github.com/apache/spark/pull/33054#issuecomment-868262286
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140304/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33054:
URL: https://github.com/apache/spark/pull/33054#issuecomment-868309951
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44839/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
HyukjinKwon edited a comment on pull request #33054:
URL: https://github.com/apache/spark/pull/33054#issuecomment-868220295
Ignoring is fine but are you sure if this is the same issue? In our case, we're importing it without try-except:
https://github.com/apache/spark/blob/master/python/pyspark/pandas/plot/core.py#L23
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33054:
URL: https://github.com/apache/spark/pull/33054#issuecomment-867421659
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140249/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33054:
URL: https://github.com/apache/spark/pull/33054#issuecomment-867411977
**[Test build #140249 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140249/testReport)** for PR 33054 at commit [`32c24d7`](https://github.com/apache/spark/commit/32c24d7f8a7652dbc03457d9b1963fbc03354a88).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33054:
URL: https://github.com/apache/spark/pull/33054#issuecomment-868267572
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44833/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33054:
URL: https://github.com/apache/spark/pull/33054#issuecomment-868202158
**[Test build #140302 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140302/testReport)** for PR 33054 at commit [`658f80d`](https://github.com/apache/spark/commit/658f80d45df7333c3ad0d7d137e02bce96735a1b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33054:
URL: https://github.com/apache/spark/pull/33054#issuecomment-868265537
**[Test build #140308 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140308/testReport)** for PR 33054 at commit [`b92032f`](https://github.com/apache/spark/commit/b92032f4dc0b5d24ae1065fbb91a3ffadd24bf4d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33054:
URL: https://github.com/apache/spark/pull/33054#issuecomment-867461504
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44778/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33054:
URL: https://github.com/apache/spark/pull/33054#issuecomment-868130771
**[Test build #140294 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140294/testReport)** for PR 33054 at commit [`0566b91`](https://github.com/apache/spark/commit/0566b91c37571297be6e2795145596aed5be274a).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33054:
URL: https://github.com/apache/spark/pull/33054#issuecomment-867454801
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44778/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33054:
URL: https://github.com/apache/spark/pull/33054#issuecomment-868291724
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #33054:
URL: https://github.com/apache/spark/pull/33054
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33054:
URL: https://github.com/apache/spark/pull/33054#issuecomment-868147214
**[Test build #140292 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140292/testReport)** for PR 33054 at commit [`b162be4`](https://github.com/apache/spark/commit/b162be4cff45f2ced8c039dd63db46e1f9107dab).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33054:
URL: https://github.com/apache/spark/pull/33054#issuecomment-868163537
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44825/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33054:
URL: https://github.com/apache/spark/pull/33054#discussion_r658671949
##########
File path: python/pyspark/pandas/plot/core.py
##########
@@ -20,7 +20,7 @@
import pandas as pd
import numpy as np
from pyspark.ml.feature import Bucketizer
-from pyspark.mllib.stat import KernelDensity
+from pyspark.mllib.stat.KernelDensity import KernelDensity
Review comment:
Can you explain this change? It looks weird. Is it a nested class?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
itholic commented on pull request #33054:
URL: https://github.com/apache/spark/pull/33054#issuecomment-868201074
Seems like it's a known bug in mypy (https://github.com/python/mypy/issues/1153).
I'd ignore this case for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33054:
URL: https://github.com/apache/spark/pull/33054#discussion_r658411219
##########
File path: python/docs/source/reference/pyspark.sql.rst
##########
@@ -218,6 +218,7 @@ DataFrame APIs
DataFrame.write
DataFrame.writeStream
DataFrame.writeTo
+ DataFrame.to_pandas_on_spark
Review comment:
can you remove the one in pyspark.pandas?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33054:
URL: https://github.com/apache/spark/pull/33054#issuecomment-867421422
**[Test build #140249 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140249/testReport)** for PR 33054 at commit [`32c24d7`](https://github.com/apache/spark/commit/32c24d7f8a7652dbc03457d9b1963fbc03354a88).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33054:
URL: https://github.com/apache/spark/pull/33054#issuecomment-868259675
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44835/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33054:
URL: https://github.com/apache/spark/pull/33054#issuecomment-868251072
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44833/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on pull request #33054: [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame
Posted by GitBox <gi...@apache.org>.
itholic commented on pull request #33054:
URL: https://github.com/apache/spark/pull/33054#issuecomment-868264170
> Ignoring is fine but are you sure if this is the same issue? In our case, we're importing it without try-except:
>
> https://github.com/apache/spark/blob/master/python/pyspark/pandas/plot/core.py#L23
Yeah, it was different one, just fixed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org