You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2021/06/24 05:48:38 UTC
[spark] branch master updated: [SPARK-35696][PYTHON][DOCS] Refine
the code examples in pandas-on-Spark documentation
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 712ed87 [SPARK-35696][PYTHON][DOCS] Refine the code examples in pandas-on-Spark documentation
712ed87 is described below
commit 712ed87faa625401a8ef546f0433ce92967d4fb5
Author: itholic <ha...@databricks.com>
AuthorDate: Thu Jun 24 14:48:13 2021 +0900
[SPARK-35696][PYTHON][DOCS] Refine the code examples in pandas-on-Spark documentation
### What changes were proposed in this pull request?
This PR proposes to refine the code examples for pandas-on-Spark since some of them still follows the naming for Koalas.
For example,
```python
kdf = ks.DataFrame({"A": [1, 2, 3], "B": [4, 5, 6]})
```
should be refined to
```python
psdf = ps.DataFrame({"A": [1, 2, 3], "B": [4, 5, 6]})
```
Also fixed the several remaining Koalas stuffs in FAQ
### Why are the changes needed?
Because we don't want to use the name "Koalas" in the Apache Spark anymore.
### Does this PR introduce _any_ user-facing change?
Yes, the examples in the documentation will be changed with refined names.
### How was this patch tested?
Manually built the docs and check one by one.
Closes #33017 from itholic/SPARK-35696.
Authored-by: itholic <ha...@databricks.com>
Signed-off-by: Hyukjin Kwon <gu...@apache.org>
---
python/docs/source/reference/pyspark.pandas/ml.rst | 4 -
.../user_guide/pandas_on_spark/best_practices.rst | 74 ++++++------
.../docs/source/user_guide/pandas_on_spark/faq.rst | 31 +++--
.../user_guide/pandas_on_spark/from_to_dbms.rst | 8 +-
.../source/user_guide/pandas_on_spark/options.rst | 126 ++++++++++-----------
.../user_guide/pandas_on_spark/pandas_pyspark.rst | 22 ++--
.../user_guide/pandas_on_spark/transform_apply.rst | 48 ++++----
.../user_guide/pandas_on_spark/typehints.rst | 20 ++--
.../source/user_guide/pandas_on_spark/types.rst | 32 +++---
9 files changed, 178 insertions(+), 187 deletions(-)
diff --git a/python/docs/source/reference/pyspark.pandas/ml.rst b/python/docs/source/reference/pyspark.pandas/ml.rst
index 272ee0c..07cdf86 100644
--- a/python/docs/source/reference/pyspark.pandas/ml.rst
+++ b/python/docs/source/reference/pyspark.pandas/ml.rst
@@ -16,10 +16,6 @@ tensorflow, ...). See comprehensive examples in
.. note::
The MLflow package must be installed in order to use this module.
- If MLflow is not installed in your environment already, you
- can install it with the following command:
-
- **pip install koalas[mlflow]**
.. autosummary::
:toctree: api/
diff --git a/python/docs/source/user_guide/pandas_on_spark/best_practices.rst b/python/docs/source/user_guide/pandas_on_spark/best_practices.rst
index 6bd0e87..4d90ed5 100644
--- a/python/docs/source/user_guide/pandas_on_spark/best_practices.rst
+++ b/python/docs/source/user_guide/pandas_on_spark/best_practices.rst
@@ -24,7 +24,7 @@ For example, if you want to configure the executor memory in Spark, you can do a
# Pandas API on Spark automatically uses this Spark context with the configurations set.
SparkContext(conf=conf)
- import pyspark.pandas as ks
+ import pyspark.pandas as ps
...
Another common configuration might be Arrow optimization in PySpark. In case of SQL configuration,
@@ -38,7 +38,7 @@ it can be set into Spark session as below:
# Pandas API on Spark automatically uses this Spark session with the configurations set.
builder.getOrCreate()
- import pyspark.pandas as ks
+ import pyspark.pandas as ps
...
All Spark features such as history server, web UI and deployment modes can be used as are with pandas API on Spark.
@@ -53,10 +53,10 @@ before the actual computation since pandas API on Spark is based on lazy executi
.. code-block:: python
- >>> import pyspark.pandas as ks
- >>> kdf = ks.DataFrame({'id': range(10)})
- >>> kdf = kdf[kdf.id > 5]
- >>> kdf.spark.explain()
+ >>> import pyspark.pandas as ps
+ >>> psdf = ps.DataFrame({'id': range(10)})
+ >>> psdf = psdf[psdf.id > 5]
+ >>> psdf.spark.explain()
== Physical Plan ==
*(1) Filter (id#1L > 5)
+- *(1) Scan ExistingRDD[__index_level_0__#0L,id#1L]
@@ -78,12 +78,12 @@ or ``DataFrame.spark.local_checkpoint()`` would be helpful.
.. code-block:: python
- >>> import pyspark.pandas as ks
- >>> kdf = ks.DataFrame({'id': range(10)})
- >>> kdf = kdf[kdf.id > 5]
- >>> kdf['id'] = kdf['id'] + (10 * kdf['id'] + kdf['id'])
- >>> kdf = kdf.groupby('id').head(2)
- >>> kdf.spark.explain()
+ >>> import pyspark.pandas as ps
+ >>> psdf = ps.DataFrame({'id': range(10)})
+ >>> psdf = psdf[psdf.id > 5]
+ >>> psdf['id'] = psdf['id'] + (10 * psdf['id'] + psdf['id'])
+ >>> psdf = psdf.groupby('id').head(2)
+ >>> psdf.spark.explain()
== Physical Plan ==
*(3) Project [__index_level_0__#0L, id#31L]
+- *(3) Filter (isnotnull(__row_number__#44) AND (__row_number__#44 <= 2))
@@ -95,8 +95,8 @@ or ``DataFrame.spark.local_checkpoint()`` would be helpful.
+- *(1) Filter (id#1L > 5)
+- *(1) Scan ExistingRDD[__index_level_0__#0L,id#1L]
- >>> kdf = kdf.spark.local_checkpoint() # or kdf.spark.checkpoint()
- >>> kdf.spark.explain()
+ >>> psdf = psdf.spark.local_checkpoint() # or psdf.spark.checkpoint()
+ >>> psdf.spark.explain()
== Physical Plan ==
*(1) Project [__index_level_0__#0L, id#31L]
+- *(1) Scan ExistingRDD[__index_level_0__#0L,id#31L,__natural_order__#59L]
@@ -115,9 +115,9 @@ and exchange the data across multiple nodes via networks. See the example below.
.. code-block:: python
- >>> import pyspark.pandas as ks
- >>> kdf = ks.DataFrame({'id': range(10)}).sort_values(by="id")
- >>> kdf.spark.explain()
+ >>> import pyspark.pandas as ps
+ >>> psdf = ps.DataFrame({'id': range(10)}).sort_values(by="id")
+ >>> psdf.spark.explain()
== Physical Plan ==
*(2) Sort [id#9L ASC NULLS LAST], true, 0
+- Exchange rangepartitioning(id#9L ASC NULLS LAST, 200), true, [id=#18]
@@ -130,16 +130,16 @@ Avoid computation on single partition
-------------------------------------
Another common case is the computation on a single partition. Currently, some APIs such as
-`DataFrame.rank <https://koalas.readthedocs.io/en/latest/reference/api/pyspark.pandas.DataFrame.rank.html>`_
+`DataFrame.rank <https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.pandas.DataFrame.rank.html>`_
uses PySpark’s Window without specifying partition specification. This leads to move all data into a single
partition in single machine and could cause serious performance degradation.
Such APIs should be avoided very large dataset.
.. code-block:: python
- >>> import pyspark.pandas as ks
- >>> kdf = ks.DataFrame({'id': range(10)})
- >>> kdf.rank().spark.explain()
+ >>> import pyspark.pandas as ps
+ >>> psdf = ps.DataFrame({'id': range(10)})
+ >>> psdf.rank().spark.explain()
== Physical Plan ==
*(4) Project [__index_level_0__#16L, id#24]
+- Window [avg(cast(_w0#26 as bigint)) windowspecdefinition(id#17L, specifiedwindowframe(RowFrame, unboundedpreceding$(), unboundedfollowing$())) AS id#24], [id#17L]
@@ -150,7 +150,7 @@ Such APIs should be avoided very large dataset.
+- *(1) Scan ExistingRDD[__index_level_0__#16L,id#17L]
Instead, use
-`GroupBy.rank <https://koalas.readthedocs.io/en/latest/reference/api/pyspark.pandas.groupby.GroupBy.rank.html>`_
+`GroupBy.rank <https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.pandas.groupby.GroupBy.rank.html>`_
as it is less expensive because data can be distributed and computed for each group.
@@ -169,9 +169,9 @@ this behavior. For instance, see below:
.. code-block:: python
- >>> import pyspark.pandas as ks
- >>> kdf = ks.DataFrame({'a': [1, 2], 'b':[3, 4]})
- >>> kdf.columns = ["a", "a"]
+ >>> import pyspark.pandas as ps
+ >>> psdf = ps.DataFrame({'a': [1, 2], 'b':[3, 4]})
+ >>> psdf.columns = ["a", "a"]
...
Reference 'a' is ambiguous, could be: a, a.;
@@ -179,8 +179,8 @@ Additionally, it is strongly discouraged to use case sensitive column names. Pan
.. code-block:: python
- >>> import pyspark.pandas as ks
- >>> kdf = ks.DataFrame({'a': [1, 2], 'A':[3, 4]})
+ >>> import pyspark.pandas as ps
+ >>> psdf = ps.DataFrame({'a': [1, 2], 'A':[3, 4]})
...
Reference 'a' is ambiguous, could be: a, a.;
@@ -193,9 +193,9 @@ However, you can turn on ``spark.sql.caseSensitive`` in Spark configuration to e
>>> builder = builder.config("spark.sql.caseSensitive", "true")
>>> builder.getOrCreate()
- >>> import pyspark.pandas as ks
- >>> kdf = ks.DataFrame({'a': [1, 2], 'A':[3, 4]})
- >>> kdf
+ >>> import pyspark.pandas as ps
+ >>> psdf = ps.DataFrame({'a': [1, 2], 'A':[3, 4]})
+ >>> psdf
a A
0 1 3
1 2 4
@@ -261,12 +261,12 @@ The examples above can be converted as below:
.. code-block:: python
- >>> import pyspark.pandas as ks
- >>> ks.Series([1, 2, 3]).max()
+ >>> import pyspark.pandas as ps
+ >>> ps.Series([1, 2, 3]).max()
3
- >>> ks.Series([1, 2, 3]).min()
+ >>> ps.Series([1, 2, 3]).min()
1
- >>> ks.Series([1, 2, 3]).sum()
+ >>> ps.Series([1, 2, 3]).sum()
6
Another common pattern from pandas users might be to rely on list comprehension or generator expression.
@@ -296,17 +296,17 @@ The example above can be also changed to directly using pandas-on-Spark APIs as
.. code-block:: python
- >>> import pyspark.pandas as ks
+ >>> import pyspark.pandas as ps
>>> import numpy as np
>>> countries = ['London', 'New York', 'Helsinki']
- >>> kser = ks.Series([20., 21., 12.], index=countries)
+ >>> psser = ps.Series([20., 21., 12.], index=countries)
>>> def square(temperature) -> np.float64:
... assert temperature > 0
... if temperature > 1000:
... temperature = None
... return temperature ** 2
...
- >>> kser.apply(square)
+ >>> psser.apply(square)
London 400.0
New York 441.0
Helsinki 144.0
diff --git a/python/docs/source/user_guide/pandas_on_spark/faq.rst b/python/docs/source/user_guide/pandas_on_spark/faq.rst
index 231be10..676e3e1 100644
--- a/python/docs/source/user_guide/pandas_on_spark/faq.rst
+++ b/python/docs/source/user_guide/pandas_on_spark/faq.rst
@@ -2,30 +2,25 @@
FAQ
===
-Is it Koalas or koalas?
------------------------
+Should I use PySpark's DataFrame API or pandas API on Spark?
+------------------------------------------------------------
-It's Koalas. Unlike pandas, we use upper case here.
+If you are already familiar with pandas and want to leverage Spark for big data, we recommend using pandas API on Spark.
+If you are learning Spark from ground up, we recommend you start with PySpark's API.
-Should I use PySpark's DataFrame API or Koalas?
------------------------------------------------
-
-If you are already familiar with pandas and want to leverage Spark for big data, we recommend
-using Koalas. If you are learning Spark from ground up, we recommend you start with PySpark's API.
-
-Does Koalas support Structured Streaming?
+Does pandas API on Spark support Structured Streaming?
-----------------------------------------
-No, Koalas does not support Structured Streaming officially.
+No, pandas API on Spark does not support Structured Streaming officially.
-As a workaround, you can use Koalas APIs with `foreachBatch` in Structured Streaming which allows batch APIs:
+As a workaround, you can use pandas-on-Spark APIs with `foreachBatch` in Structured Streaming which allows batch APIs:
.. code-block:: python
>>> def func(batch_df, batch_id):
- ... koalas_df = ks.DataFrame(batch_df)
- ... koalas_df['a'] = 1
- ... print(koalas_df)
+ ... pandas_on_spark_df = ps.DataFrame(batch_df)
+ ... pandas_on_spark_df['a'] = 1
+ ... print(pandas_on_spark_df)
>>> spark.readStream.format("rate").load().writeStream.foreachBatch(func).start()
timestamp value a
@@ -34,11 +29,11 @@ As a workaround, you can use Koalas APIs with `foreachBatch` in Structured Strea
0 2020-02-21 09:49:38.574 5 1
...
-How is Koalas different from Dask?
-----------------------------------
+How is pandas API on Spark different from Dask?
+-----------------------------------------------
Different projects have different focuses. Spark is already deployed in virtually every
organization, and often is the primary interface to the massive amount of data stored in data lakes.
-Koalas was inspired by Dask, and aims to make the transition from pandas to Spark easy for data
+pandas API on Spark was inspired by Dask, and aims to make the transition from pandas to Spark easy for data
scientists.
diff --git a/python/docs/source/user_guide/pandas_on_spark/from_to_dbms.rst b/python/docs/source/user_guide/pandas_on_spark/from_to_dbms.rst
index 1f9e794..429bdcd 100644
--- a/python/docs/source/user_guide/pandas_on_spark/from_to_dbms.rst
+++ b/python/docs/source/user_guide/pandas_on_spark/from_to_dbms.rst
@@ -22,7 +22,7 @@ pandas-on-Spark needs a canonical JDBC URL for ``con``, and is able to take extr
.. code-block:: python
- ks.read_sql(..., dbtable="...", driver="", keytab="", ...)
+ ps.read_sql(..., dbtable="...", driver="", keytab="", ...)
Reading and writing DataFrames
@@ -77,9 +77,9 @@ Now, you're ready to read the table:
.. code-block:: python
- import pyspark.pandas as ks
+ import pyspark.pandas as ps
- df = ks.read_sql("stocks", con="jdbc:sqlite:{}/example.db".format(os.getcwd()))
+ df = ps.read_sql("stocks", con="jdbc:sqlite:{}/example.db".format(os.getcwd()))
df
.. code-block:: text
@@ -98,7 +98,7 @@ You can also write it back to the ``stocks`` table as below:
df.to_spark_io(
format="jdbc", mode="append",
dbtable="stocks", url="jdbc:sqlite:{}/example.db".format(os.getcwd()))
- ks.read_sql("stocks", con="jdbc:sqlite:{}/example.db".format(os.getcwd()))
+ ps.read_sql("stocks", con="jdbc:sqlite:{}/example.db".format(os.getcwd()))
.. code-block:: text
diff --git a/python/docs/source/user_guide/pandas_on_spark/options.rst b/python/docs/source/user_guide/pandas_on_spark/options.rst
index 66b7a93..847daa4 100644
--- a/python/docs/source/user_guide/pandas_on_spark/options.rst
+++ b/python/docs/source/user_guide/pandas_on_spark/options.rst
@@ -12,28 +12,28 @@ You can get/set options directly as attributes of the top-level ``options`` attr
.. code-block:: python
- >>> import pyspark.pandas as ks
- >>> ks.options.display.max_rows
+ >>> import pyspark.pandas as ps
+ >>> ps.options.display.max_rows
1000
- >>> ks.options.display.max_rows = 10
- >>> ks.options.display.max_rows
+ >>> ps.options.display.max_rows = 10
+ >>> ps.options.display.max_rows
10
-The API is composed of 3 relevant functions, available directly from the ``koalas``
+The API is composed of 3 relevant functions, available directly from the ``pandas_on_spark``
namespace:
* :func:`get_option` / :func:`set_option` - get/set the value of a single option.
* :func:`reset_option` - reset one or more options to their default value.
-**Note:** Developers can check out `pyspark.pandas/config.py <https://github.com/databricks/koalas/blob/master/databricks/koalas/config.py>`_ for more information.
+**Note:** Developers can check out `pyspark.pandas/config.py <https://github.com/apache/spark/blob/master/python/pyspark/pandas/config.py>`_ for more information.
.. code-block:: python
- >>> import pyspark.pandas as ks
- >>> ks.get_option("display.max_rows")
+ >>> import pyspark.pandas as ps
+ >>> ps.get_option("display.max_rows")
1000
- >>> ks.set_option("display.max_rows", 101)
- >>> ks.get_option("display.max_rows")
+ >>> ps.set_option("display.max_rows", 101)
+ >>> ps.get_option("display.max_rows")
101
@@ -41,35 +41,35 @@ Getting and setting options
---------------------------
As described above, :func:`get_option` and :func:`set_option`
-are available from the koalas namespace. To change an option, call
+are available from the ``pandas_on_spark`` namespace. To change an option, call
``set_option('option name', new_value)``.
.. code-block:: python
- >>> import pyspark.pandas as ks
- >>> ks.get_option('compute.max_rows')
+ >>> import pyspark.pandas as ps
+ >>> ps.get_option('compute.max_rows')
1000
- >>> ks.set_option('compute.max_rows', 2000)
- >>> ks.get_option('compute.max_rows')
+ >>> ps.set_option('compute.max_rows', 2000)
+ >>> ps.get_option('compute.max_rows')
2000
All options also have a default value, and you can use ``reset_option`` to do just that:
.. code-block:: python
- >>> import pyspark.pandas as ks
- >>> ks.reset_option("display.max_rows")
+ >>> import pyspark.pandas as ps
+ >>> ps.reset_option("display.max_rows")
.. code-block:: python
- >>> import pyspark.pandas as ks
- >>> ks.get_option("display.max_rows")
+ >>> import pyspark.pandas as ps
+ >>> ps.get_option("display.max_rows")
1000
- >>> ks.set_option("display.max_rows", 999)
- >>> ks.get_option("display.max_rows")
+ >>> ps.set_option("display.max_rows", 999)
+ >>> ps.get_option("display.max_rows")
999
- >>> ks.reset_option("display.max_rows")
- >>> ks.get_option("display.max_rows")
+ >>> ps.reset_option("display.max_rows")
+ >>> ps.get_option("display.max_rows")
1000
``option_context`` context manager has been exposed through
@@ -78,13 +78,13 @@ are restored automatically when you exit the `with` block:
.. code-block:: python
- >>> with ks.option_context("display.max_rows", 10, "compute.max_rows", 5):
- ... print(ks.get_option("display.max_rows"))
- ... print(ks.get_option("compute.max_rows"))
+ >>> with ps.option_context("display.max_rows", 10, "compute.max_rows", 5):
+ ... print(ps.get_option("display.max_rows"))
+ ... print(ps.get_option("compute.max_rows"))
10
5
- >>> print(ks.get_option("display.max_rows"))
- >>> print(ks.get_option("compute.max_rows"))
+ >>> print(ps.get_option("display.max_rows"))
+ >>> print(ps.get_option("compute.max_rows"))
1000
1000
@@ -100,35 +100,35 @@ See the examples below.
.. code-block:: python
- >>> import pyspark.pandas as ks
- >>> ks.set_option('compute.ops_on_diff_frames', True)
- >>> kdf1 = ks.range(5)
- >>> kdf2 = ks.DataFrame({'id': [5, 4, 3]})
- >>> (kdf1 - kdf2).sort_index()
+ >>> import pyspark.pandas as ps
+ >>> ps.set_option('compute.ops_on_diff_frames', True)
+ >>> psdf1 = ps.range(5)
+ >>> psdf2 = ps.DataFrame({'id': [5, 4, 3]})
+ >>> (psdf1 - psdf2).sort_index()
id
0 -5.0
1 -3.0
2 -1.0
3 NaN
4 NaN
- >>> ks.reset_option('compute.ops_on_diff_frames')
+ >>> ps.reset_option('compute.ops_on_diff_frames')
.. code-block:: python
- >>> import pyspark.pandas as ks
- >>> ks.set_option('compute.ops_on_diff_frames', True)
- >>> kdf = ks.range(5)
- >>> kser_a = ks.Series([1, 2, 3, 4])
- >>> # 'kser_a' is not from 'kdf' DataFrame. So it is considered as a Series not from 'kdf'.
- >>> kdf['new_col'] = kser_a
- >>> kdf
+ >>> import pyspark.pandas as ps
+ >>> ps.set_option('compute.ops_on_diff_frames', True)
+ >>> psdf = ps.range(5)
+ >>> psser_a = ps.Series([1, 2, 3, 4])
+ >>> # 'psser_a' is not from 'psdf' DataFrame. So it is considered as a Series not from 'psdf'.
+ >>> psdf['new_col'] = psser_a
+ >>> psdf
id new_col
0 0 1.0
1 1 2.0
3 3 4.0
2 2 3.0
4 4 NaN
- >>> ks.reset_option('compute.ops_on_diff_frames')
+ >>> ps.reset_option('compute.ops_on_diff_frames')
Default Index type
@@ -146,11 +146,11 @@ This index type should be avoided when the data is large. This is default. See t
.. code-block:: python
- >>> import pyspark.pandas as ks
- >>> ks.set_option('compute.default_index_type', 'sequence')
- >>> kdf = ks.range(3)
- >>> ks.reset_option('compute.default_index_type')
- >>> kdf.index
+ >>> import pyspark.pandas as ps
+ >>> ps.set_option('compute.default_index_type', 'sequence')
+ >>> psdf = ps.range(3)
+ >>> ps.reset_option('compute.default_index_type')
+ >>> psdf.index
Int64Index([0, 1, 2], dtype='int64')
This is conceptually equivalent to the PySpark example as below:
@@ -158,8 +158,8 @@ This is conceptually equivalent to the PySpark example as below:
.. code-block:: python
>>> from pyspark.sql import functions as F, Window
- >>> import pyspark.pandas as ks
- >>> spark_df = ks.range(3).to_spark()
+ >>> import pyspark.pandas as ps
+ >>> spark_df = ps.range(3).to_spark()
>>> sequential_index = F.row_number().over(
... Window.orderBy(F.monotonically_increasing_id().asc())) - 1
>>> spark_df.select(sequential_index).rdd.map(lambda r: r[0]).collect()
@@ -174,19 +174,19 @@ then it does not guarantee the sequential index. See the example below:
.. code-block:: python
- >>> import pyspark.pandas as ks
- >>> ks.set_option('compute.default_index_type', 'distributed-sequence')
- >>> kdf = ks.range(3)
- >>> ks.reset_option('compute.default_index_type')
- >>> kdf.index
+ >>> import pyspark.pandas as ps
+ >>> ps.set_option('compute.default_index_type', 'distributed-sequence')
+ >>> psdf = ps.range(3)
+ >>> ps.reset_option('compute.default_index_type')
+ >>> psdf.index
Int64Index([0, 1, 2], dtype='int64')
This is conceptually equivalent to the PySpark example as below:
.. code-block:: python
- >>> import pyspark.pandas as ks
- >>> spark_df = ks.range(3).to_spark()
+ >>> import pyspark.pandas as ps
+ >>> spark_df = ps.range(3).to_spark()
>>> spark_df.rdd.zipWithIndex().map(lambda p: p[1]).collect()
[0, 1, 2]
@@ -198,11 +198,11 @@ have any penalty comparing to other index types. See the example below:
.. code-block:: python
- >>> import pyspark.pandas as ks
- >>> ks.set_option('compute.default_index_type', 'distributed')
- >>> kdf = ks.range(3)
- >>> ks.reset_option('compute.default_index_type')
- >>> kdf.index
+ >>> import pyspark.pandas as ps
+ >>> ps.set_option('compute.default_index_type', 'distributed')
+ >>> psdf = ps.range(3)
+ >>> ps.reset_option('compute.default_index_type')
+ >>> psdf.index
Int64Index([25769803776, 60129542144, 94489280512], dtype='int64')
This is conceptually equivalent to the PySpark example as below:
@@ -210,8 +210,8 @@ This is conceptually equivalent to the PySpark example as below:
.. code-block:: python
>>> from pyspark.sql import functions as F
- >>> import pyspark.pandas as ks
- >>> spark_df = ks.range(3).to_spark()
+ >>> import pyspark.pandas as ps
+ >>> spark_df = ps.range(3).to_spark()
>>> spark_df.select(F.monotonically_increasing_id()) \
... .rdd.map(lambda r: r[0]).collect()
[25769803776, 60129542144, 94489280512]
diff --git a/python/docs/source/user_guide/pandas_on_spark/pandas_pyspark.rst b/python/docs/source/user_guide/pandas_on_spark/pandas_pyspark.rst
index 6796cbb..b6b433a 100644
--- a/python/docs/source/user_guide/pandas_on_spark/pandas_pyspark.rst
+++ b/python/docs/source/user_guide/pandas_on_spark/pandas_pyspark.rst
@@ -23,10 +23,10 @@ as below:
.. code-block:: python
- >>> import pyspark.pandas as ks
+ >>> import pyspark.pandas as ps
>>>
- >>> kdf = ks.range(10)
- >>> pdf = kdf.to_pandas()
+ >>> psdf = ps.range(10)
+ >>> pdf = psdf.to_pandas()
>>> pdf.values
array([[0],
[1],
@@ -43,7 +43,7 @@ pandas DataFrame can be a pandas-on-Spark DataFrame easily as below:
.. code-block:: python
- >>> ks.from_pandas(pdf)
+ >>> ps.from_pandas(pdf)
id
0 0
1 1
@@ -71,10 +71,10 @@ as below:
.. code-block:: python
- >>> import pyspark.pandas as ks
+ >>> import pyspark.pandas as ps
>>>
- >>> kdf = ks.range(10)
- >>> sdf = kdf.to_spark().filter("id > 5")
+ >>> psdf = ps.range(10)
+ >>> sdf = psdf.to_spark().filter("id > 5")
>>> sdf.show()
+---+
| id|
@@ -89,7 +89,7 @@ Spark DataFrame can be a pandas-on-Spark DataFrame easily as below:
.. code-block:: python
- >>> sdf.to_koalas()
+ >>> sdf.to_pandas_on_spark()
id
0 6
1 7
@@ -103,13 +103,13 @@ to use as an index when possible.
.. code-block:: python
>>> # Create a pandas-on-Spark DataFrame with an explicit index.
- ... kdf = ks.DataFrame({'id': range(10)}, index=range(10))
+ ... psdf = ps.DataFrame({'id': range(10)}, index=range(10))
>>> # Keep the explicit index.
- ... sdf = kdf.to_spark(index_col='index')
+ ... sdf = psdf.to_spark(index_col='index')
>>> # Call Spark APIs
... sdf = sdf.filter("id > 5")
>>> # Uses the explicit index to avoid to create default index.
- ... sdf.to_koalas(index_col='index')
+ ... sdf.to_pandas_on_spark(index_col='index')
id
index
6 6
diff --git a/python/docs/source/user_guide/pandas_on_spark/transform_apply.rst b/python/docs/source/user_guide/pandas_on_spark/transform_apply.rst
index 46d5980..6e44156 100644
--- a/python/docs/source/user_guide/pandas_on_spark/transform_apply.rst
+++ b/python/docs/source/user_guide/pandas_on_spark/transform_apply.rst
@@ -7,8 +7,8 @@ Transform and apply a function
.. currentmodule:: pyspark.pandas
There are many APIs that allow users to apply a function against pandas-on-Spark DataFrame such as
-:func:`DataFrame.transform`, :func:`DataFrame.apply`, :func:`DataFrame.koalas.transform_batch`,
-:func:`DataFrame.koalas.apply_batch`, :func:`Series.koalas.transform_batch`, etc. Each has a distinct
+:func:`DataFrame.transform`, :func:`DataFrame.apply`, :func:`DataFrame.pandas_on_spark.transform_batch`,
+:func:`DataFrame.pandas_on_spark.apply_batch`, :func:`Series.pandas_on_spark.transform_batch`, etc. Each has a distinct
purpose and works differently internally. This section describes the differences among
them where users are confused often.
@@ -20,19 +20,19 @@ to return the same length of the input and the latter does not require this. See
.. code-block:: python
- >>> kdf = ks.DataFrame({'a': [1,2,3], 'b':[4,5,6]})
+ >>> psdf = ps.DataFrame({'a': [1,2,3], 'b':[4,5,6]})
>>> def pandas_plus(pser):
... return pser + 1 # should always return the same length as input.
...
- >>> kdf.transform(pandas_plus)
+ >>> psdf.transform(pandas_plus)
.. code-block:: python
- >>> kdf = ks.DataFrame({'a': [1,2,3], 'b':[5,6,7]})
+ >>> psdf = ps.DataFrame({'a': [1,2,3], 'b':[5,6,7]})
>>> def pandas_plus(pser):
... return pser[pser % 2 == 1] # allows an arbitrary length
...
- >>> kdf.apply(pandas_plus)
+ >>> psdf.apply(pandas_plus)
In this case, each function takes a pandas Series, and pandas API on Spark computes the functions in a distributed manner as below.
@@ -45,11 +45,11 @@ In case of 'column' axis, the function takes each row as a pandas Series.
.. code-block:: python
- >>> kdf = ks.DataFrame({'a': [1,2,3], 'b':[4,5,6]})
+ >>> psdf = ps.DataFrame({'a': [1,2,3], 'b':[4,5,6]})
>>> def pandas_plus(pser):
... return sum(pser) # allows an arbitrary length
...
- >>> kdf.apply(pandas_plus, axis='columns')
+ >>> psdf.apply(pandas_plus, axis='columns')
The example above calculates the summation of each row as a pandas Series. See below:
@@ -62,59 +62,59 @@ In the examples above, the type hints were not used for simplicity but it is enc
Please refer the API documentations.
-``koalas.transform_batch`` and ``koalas.apply_batch``
------------------------------------------------------
+``pandas_on_spark.transform_batch`` and ``pandas_on_spark.apply_batch``
+-----------------------------------------------------------------------
-In :func:`DataFrame.koalas.transform_batch`, :func:`DataFrame.koalas.apply_batch`, :func:`Series.koalas.transform_batch`, etc., the ``batch``
+In :func:`DataFrame.pandas_on_spark.transform_batch`, :func:`DataFrame.pandas_on_spark.apply_batch`, :func:`Series.pandas_on_spark.transform_batch`, etc., the ``batch``
postfix means each chunk in pandas-on-Spark DataFrame or Series. The APIs slice the pandas-on-Spark DataFrame or Series, and
then applies the given function with pandas DataFrame or Series as input and output. See the examples below:
.. code-block:: python
- >>> kdf = ks.DataFrame({'a': [1,2,3], 'b':[4,5,6]})
+ >>> psdf = ps.DataFrame({'a': [1,2,3], 'b':[4,5,6]})
>>> def pandas_plus(pdf):
... return pdf + 1 # should always return the same length as input.
...
- >>> kdf.koalas.transform_batch(pandas_plus)
+ >>> psdf.pandas_on_spark.transform_batch(pandas_plus)
.. code-block:: python
- >>> kdf = ks.DataFrame({'a': [1,2,3], 'b':[4,5,6]})
+ >>> psdf = ps.DataFrame({'a': [1,2,3], 'b':[4,5,6]})
>>> def pandas_plus(pdf):
... return pdf[pdf.a > 1] # allow arbitrary length
...
- >>> kdf.koalas.apply_batch(pandas_plus)
+ >>> psdf.pandas_on_spark.apply_batch(pandas_plus)
The functions in both examples take a pandas DataFrame as a chunk of pandas-on-Spark DataFrame, and output a pandas DataFrame.
Pandas API on Spark combines the pandas DataFrames as a pandas-on-Spark DataFrame.
-Note that :func:`DataFrame.koalas.transform_batch` has the length restriction - the length of input and output should be
-the same whereas :func:`DataFrame.koalas.apply_batch` does not. However, it is important to know that
-the output belongs to the same DataFrame when :func:`DataFrame.koalas.transform_batch` returns a Series, and
-you can avoid a shuffle by the operations between different DataFrames. In case of :func:`DataFrame.koalas.apply_batch`, its output is always
+Note that :func:`DataFrame.pandas_on_spark.transform_batch` has the length restriction - the length of input and output should be
+the same whereas :func:`DataFrame.pandas_on_spark.apply_batch` does not. However, it is important to know that
+the output belongs to the same DataFrame when :func:`DataFrame.pandas_on_spark.transform_batch` returns a Series, and
+you can avoid a shuffle by the operations between different DataFrames. In case of :func:`DataFrame.pandas_on_spark.apply_batch`, its output is always
treated that it belongs to a new different DataFrame. See also
`Operations on different DataFrames <options.rst#operations-on-different-dataframes>`_ for more details.
.. image:: https://user-images.githubusercontent.com/6477701/80076779-9f6cac80-8587-11ea-8c92-07d7b992733b.png
- :alt: koalas.transform_batch and koalas.apply_batch in Frame
+ :alt: pandas_on_spark.transform_batch and pandas_on_spark.apply_batch in Frame
:align: center
:width: 650
-In case of :func:`Series.koalas.transform_batch`, it is also similar with :func:`DataFrame.koalas.transform_batch`; however, it takes
+In case of :func:`Series.pandas_on_spark.transform_batch`, it is also similar with :func:`DataFrame.pandas_on_spark.transform_batch`; however, it takes
a pandas Series as a chunk of pandas-on-Spark Series.
.. code-block:: python
- >>> kdf = ks.DataFrame({'a': [1,2,3], 'b':[4,5,6]})
+ >>> psdf = ps.DataFrame({'a': [1,2,3], 'b':[4,5,6]})
>>> def pandas_plus(pser):
... return pser + 1 # should always return the same length as input.
...
- >>> kdf.a.koalas.transform_batch(pandas_plus)
+ >>> psdf.a.pandas_on_spark.transform_batch(pandas_plus)
Under the hood, each batch of pandas-on-Spark Series is split to multiple pandas Series, and each function computes on that as below:
.. image:: https://user-images.githubusercontent.com/6477701/80076795-a3003380-8587-11ea-8b73-186e4047f8c0.png
- :alt: koalas.transform_batch in Series
+ :alt: pandas_on_spark.transform_batch in Series
:width: 350
:align: center
diff --git a/python/docs/source/user_guide/pandas_on_spark/typehints.rst b/python/docs/source/user_guide/pandas_on_spark/typehints.rst
index cb12271..c7b2c5e 100644
--- a/python/docs/source/user_guide/pandas_on_spark/typehints.rst
+++ b/python/docs/source/user_guide/pandas_on_spark/typehints.rst
@@ -6,8 +6,8 @@ Type Hints in Pandas API on Spark
Pandas API on Spark, by default, infers the schema by taking some top records from the output,
in particular, when you use APIs that allow users to apply a function against pandas-on-Spark DataFrame
-such as :func:`DataFrame.transform`, :func:`DataFrame.apply`, :func:`DataFrame.koalas.apply_batch`,
-:func:`DataFrame.koalas.apply_batch`, :func:`Series.koalas.apply_batch`, etc.
+such as :func:`DataFrame.transform`, :func:`DataFrame.apply`, :func:`DataFrame.pandas_on_spark.apply_batch`,
+:func:`DataFrame.pandas_on_spark.apply_batch`, :func:`Series.pandas_on_spark.apply_batch`, etc.
However, this is potentially expensive. If there are several expensive operations such as a shuffle
in the upstream of the execution plan, pandas API on Spark will end up with executing the Spark job twice, once
@@ -35,11 +35,11 @@ it as a Spark schema. As an example, you can specify the return type hint as bel
.. code-block:: python
- >>> def pandas_div(pdf) -> ks.DataFrame[float, float]:
+ >>> def pandas_div(pdf) -> ps.DataFrame[float, float]:
... # pdf is a pandas DataFrame.
... return pdf[['B', 'C']] / pdf[['B', 'C']]
...
- >>> df = ks.DataFrame({'A': ['a', 'a', 'b'], 'B': [1, 2, 3], 'C': [4, 6, 5]})
+ >>> df = ps.DataFrame({'A': ['a', 'a', 'b'], 'B': [1, 2, 3], 'C': [4, 6, 5]})
>>> df.groupby('A').apply(pandas_div)
The function ``pandas_div`` actually takes and outputs a pandas DataFrame instead of pandas-on-Spark :class:`DataFrame`.
@@ -53,7 +53,7 @@ From pandas-on-Spark 1.0 with Python 3.7+, now you can specify the type hints by
... # pdf is a pandas DataFrame.
... return pdf[['B', 'C']] / pdf[['B', 'C']]
...
- >>> df = ks.DataFrame({'A': ['a', 'a', 'b'], 'B': [1, 2, 3], 'C': [4, 6, 5]})
+ >>> df = ps.DataFrame({'A': ['a', 'a', 'b'], 'B': [1, 2, 3], 'C': [4, 6, 5]})
>>> df.groupby('A').apply(pandas_div)
Likewise, pandas Series can be also used as a type hints:
@@ -63,7 +63,7 @@ Likewise, pandas Series can be also used as a type hints:
>>> def sqrt(x) -> pd.Series[float]:
... return np.sqrt(x)
...
- >>> df = ks.DataFrame([[4, 9]] * 3, columns=['A', 'B'])
+ >>> df = ps.DataFrame([[4, 9]] * 3, columns=['A', 'B'])
>>> df.apply(sqrt, axis=0)
Currently, both pandas API on Spark and pandas instances can be used to specify the type hints; however, pandas-on-Spark
@@ -84,7 +84,7 @@ the column names as ``c#`` and this easily leads users to lose or forgot the Ser
... pdf['A'] = pdf.id + 1
... return pdf
...
- >>> ks.range(5).koalas.apply_batch(transform)
+ >>> ps.range(5).pandas_on_spark.apply_batch(transform)
.. code-block:: bash
@@ -105,7 +105,7 @@ the Series names, ``id`` and ``A``, and ``int`` types respectively.
... pdf['A'] = pdf.id + 1
... return pdf
...
- >>> ks.range(5).koalas.apply_batch(transform)
+ >>> ps.range(5).pandas_on_spark.apply_batch(transform)
.. code-block:: bash
@@ -124,7 +124,7 @@ programmatically generate the return type and schema.
>>> def transform(pdf) -> pd.DataFrame[zip(pdf.columns, pdf.dtypes)]:
... return pdf + 1
...
- >>> kdf.koalas.apply_batch(transform)
+ >>> psdf.pandas_on_spark.apply_batch(transform)
Likewise, ``dtype`` instances from pandas DataFrame can be used alone and let pandas API on Spark generate column names.
@@ -133,4 +133,4 @@ Likewise, ``dtype`` instances from pandas DataFrame can be used alone and let pa
>>> def transform(pdf) -> pd.DataFrame[pdf.dtypes]:
... return pdf + 1
...
- >>> kdf.koalas.apply_batch(transform)
+ >>> psdf.pandas_on_spark.apply_batch(transform)
diff --git a/python/docs/source/user_guide/pandas_on_spark/types.rst b/python/docs/source/user_guide/pandas_on_spark/types.rst
index 194a0ad..7176a9c 100644
--- a/python/docs/source/user_guide/pandas_on_spark/types.rst
+++ b/python/docs/source/user_guide/pandas_on_spark/types.rst
@@ -26,10 +26,10 @@ The example below shows how data types are casted from PySpark DataFrame to pand
DataFrame[tinyint: tinyint, decimal: decimal(10,0), float: float, double: double, integer: int, long: bigint, short: smallint, timestamp: timestamp, string: string, boolean: boolean, date: date]
# 3. Convert PySpark DataFrame to pandas-on-Spark DataFrame
- >>> kdf = sdf.to_koalas()
+ >>> psdf = sdf.to_pandas_on_spark()
# 4. Check the pandas-on-Spark data types
- >>> kdf.dtypes
+ >>> psdf.dtypes
tinyint int8
decimal object
float float32
@@ -49,16 +49,16 @@ The example below shows how data types are casted from pandas-on-Spark DataFrame
.. code-block:: python
# 1. Create a pandas-on-Spark DataFrame
- >>> kdf = ks.DataFrame({"int8": [1], "bool": [True], "float32": [1.0], "float64": [1.0], "int32": [1], "int64": [1], "int16": [1], "datetime": [datetime.datetime(2020, 10, 27)], "object_string": ["1"], "object_decimal": [decimal.Decimal("1.1")], "object_date": [datetime.date(2020, 10, 27)]})
+ >>> psdf = ps.DataFrame({"int8": [1], "bool": [True], "float32": [1.0], "float64": [1.0], "int32": [1], "int64": [1], "int16": [1], "datetime": [datetime.datetime(2020, 10, 27)], "object_string": ["1"], "object_decimal": [decimal.Decimal("1.1")], "object_date": [datetime.date(2020, 10, 27)]})
# 2. Type casting by using `astype`
- >>> kdf['int8'] = kdf['int8'].astype('int8')
- >>> kdf['int16'] = kdf['int16'].astype('int16')
- >>> kdf['int32'] = kdf['int32'].astype('int32')
- >>> kdf['float32'] = kdf['float32'].astype('float32')
+ >>> psdf['int8'] = psdf['int8'].astype('int8')
+ >>> psdf['int16'] = psdf['int16'].astype('int16')
+ >>> psdf['int32'] = psdf['int32'].astype('int32')
+ >>> psdf['float32'] = psdf['float32'].astype('float32')
# 3. Check the pandas-on-Spark data types
- >>> kdf.dtypes
+ >>> psdf.dtypes
int8 int8
bool bool
float32 float32
@@ -73,7 +73,7 @@ The example below shows how data types are casted from pandas-on-Spark DataFrame
dtype: object
# 4. Convert pandas-on-Spark DataFrame to PySpark DataFrame
- >>> sdf = kdf.to_spark()
+ >>> sdf = psdf.to_spark()
# 5. Check the PySpark data types
>>> sdf
@@ -88,7 +88,7 @@ When converting pandas-on-Spark DataFrame to pandas DataFrame, and the data type
.. code-block:: python
# Convert pandas-on-Spark DataFrame to pandas DataFrame
- >>> pdf = kdf.to_pandas()
+ >>> pdf = psdf.to_pandas()
# Check the pandas data types
>>> pdf.dtypes
@@ -111,7 +111,7 @@ However, there are several data types only provided by pandas.
.. code-block:: python
# pd.Catrgorical type is not supported in pandas API on Spark yet.
- >>> ks.Series([pd.Categorical([1, 2, 3])])
+ >>> ps.Series([pd.Categorical([1, 2, 3])])
Traceback (most recent call last):
...
pyarrow.lib.ArrowInvalid: Could not convert [1, 2, 3]
@@ -201,16 +201,16 @@ You can also check the underlying PySpark data type of `Series` or schema of `Da
.. code-block:: python
- >>> ks.Series([0.3, 0.1, 0.8]).spark.data_type
+ >>> ps.Series([0.3, 0.1, 0.8]).spark.data_type
DoubleType
- >>> ks.Series(["welcome", "to", "pandas-on-Spark"]).spark.data_type
+ >>> ps.Series(["welcome", "to", "pandas-on-Spark"]).spark.data_type
StringType
- >>> ks.Series([[False, True, False]]).spark.data_type
+ >>> ps.Series([[False, True, False]]).spark.data_type
ArrayType(BooleanType,true)
- >>> ks.DataFrame({"d": [0.3, 0.1, 0.8], "s": ["welcome", "to", "pandas-on-Spark"], "b": [False, True, False]}).spark.print_schema()
+ >>> ps.DataFrame({"d": [0.3, 0.1, 0.8], "s": ["welcome", "to", "pandas-on-Spark"], "b": [False, True, False]}).spark.print_schema()
root
|-- d: double (nullable = false)
|-- s: string (nullable = false)
@@ -222,7 +222,7 @@ You can also check the underlying PySpark data type of `Series` or schema of `Da
.. code-block:: python
- >>> ks.Series([1, "A"])
+ >>> ps.Series([1, "A"])
Traceback (most recent call last):
...
TypeError: an integer is required (got type str)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org