You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2019/03/28 14:12:10 UTC
[GitHub] [spark] srowen commented on a change in pull request #24234:
[WIP][SPARK_26022][PYTHON][DOCS] PySpark Comparison with Pandas
srowen commented on a change in pull request #24234: [WIP][SPARK_26022][PYTHON][DOCS] PySpark Comparison with Pandas
URL: https://github.com/apache/spark/pull/24234#discussion_r270017611
##########
File path: docs/sql-pyspark-comparison-with-pandas.md
##########
@@ -0,0 +1,401 @@
+---
+layout: global
+title: PySpark Comparison with Pandas
+displayTitle: PySpark Comparison with Pandas
+---
+
+Both PySpark and Pandas cover important use cases and provide a rich set of features to interact
+with various structural and semistructral data in Python world. Often, PySpark users are used to
+Pandas. Therefore, this document targets to document the comparison.
+
+* Overview
+* DataFrame APIs
+ * Quick References
+ * Create DataFrame
+ * Load DataFrame
+ * Save DataFrame
+ * Inspect DataFrame
+ * Interaction between PySpark and Pandas
+* Notable Differences
+ * Lazy and Eager Evaluation
+ * Direct assignment
+ * NULL, None, NaN and NaT
+ * Type inference, coercion and cast
+
+
+## Overview
+
+PySpark and Pandas support common functionality to load, save, create, transform and describe
+DataFrame. PySpark provides conversion from/to Pandas DataFrame, and PySpark introduced Pandas
+UDFs which allow to use Pandas APIs as are for interoperability between them.
+
+Nevertheless, there are fundamental differences between them to note in general.
+
+1. PySpark DataFrame is a distributed dataset across multiple nodes whereas Pandas DataFrame is a
+ local dataset within single node.
+
+ It brings a practical point. If you handle larget dataset, arguably PySpark brings arguably a
+ better performance in general. If the dataset to process does not fix into the memory in a
+ single node, using PySpark is probably the way. In case of small dataset, Pandas might be
+ faster in general since there would not be overhead, for instance, network.
+
+2. PySpark DataFrame is lazy evaluation whereas Pandas DataFrame is eager evaluation.
+
+ PySpark DataFrame executes lazily whereas Pandas DataFrame executes each operation
+ immediately against the data set.
+
+3. PySpark DataFrame is immutable in nature whereas Pandas DataFrame is mutable.
+
+ In PySpark, it creates DataFrame once which cannot be changed. Instead, it should transform
+ it to another DataFrame whereas Pandas DataFrame is mutable which directly updates the state
+ of it. Typical example is `String` vs `StringBuilder` in Java.
+
+4. PySpark operations on DataFrame tend to comply SQL.
+
+ It causes some subtleties comparing to Pandas, for instance, about `NaN`, `None` and `NULL`.
+
+There are similarities and differences between them which might bring confusion. In this document
+these are described and illuminated by several examples.
+
+
+
+## DataFrame APIs
+
+This chapter describes DataFrame APIs in both PySpark and Pandas.
+
+
+### Quick References
+
+| PySpark | Pandas |
+| ------------------------------------------------------------------ | ---------------------------------------- |
+| `df.limit(3)` | `df.head(3)` |
+| `df.filter("a == 1 AND b == 2")` | `df.filter("(df.a == 1) & (df.b == 2)")` |
+| `df.filter((df.a == 1) & (df.b == 2))` | `df[(df.a == 1) & (df.b == 2)]` |
+| `df.select("a", "b")` | `df[["a", "b"]]` |
+| `df.drop_duplicates()` | `df.drop_duplicates()` |
+| `df.sample(fraction=0.01)` | `df.sample(frac=0.01)` |
+| `df.groupby("a").count()` | `df.groupby("a").size()` |
+| `df.groupby("a").agg({"b": "sum"})` | `df.groupby("a").agg({"b": np.sum})` |
+| `df1.join(df2, on="a")` | `pandas.merge(df1, df2, on="a")` |
+| `df1.union(df2)` | `pandas.concat(df1, df2)` |
+| `df = df.select(when(df["a"] < 5, df["a"] * 2).otherwise(df["a"]))`| `df.loc[pdf['a'] < 5, 'a'] *= 2` |
+
+
+### Create DataFrame
+
+In order to create DataFrame in PySpark and Pandas, you can run the codes below:
+
+```python
+# PySpark
+data = zip(['Chicago', 'San Francisco', 'New York City'], range(1, 4))
+spark.createDataFrame(list(data), ["city", "rank"])
+```
+
+```python
+# Pandas
+data = {'city': ['Chicago', 'San Francisco', 'New York City'], 'rank': range(1, 4)}
+pandas.DataFrame(data)
+```
+
+One notable difference when creating DataFrame is that Pandas accepts the data as below:
+
+```
+data = {
+ 'city': ['Chicago', 'San Francisco', 'New York City'],
+ 'rank': range(1, 4)
+}
+```
+
+and it interprets as:
+
+```
+ city rank
+0 Chicago 1
+1 San Francisco 2
+2 New York City 3
+```
+
+So, a dictionary that contains key and multiple values becomes DataFrame but PySpark does
+support this. Instead, PySpark can make DataFrame from Pandas DataFrame as below:
+
+```
+createDataFrame(pandas.DataFrame(data))
+```
+
+For more information see "Interaction between PySpark and Pandas" below.
+
+
+### Load DataFrame
+
+To load DataFrame, there are APIs, usually, `spark.read.xxx` (or `spark.read.format("xxx")`)
+for PySpark and `pandas.read_xxx` for Pandas.
+
+```python
+# PySpark
+df = spark.read.csv("data.csv")
+```
+
+```python
+# Pandas
+df = pandas.read_csv("data.csv")
+```
+
+There are many sources available in both PySpark and Pandas. For example, CSV, JSON and
+Parquet are commonly supported but each supports different set of sources. There are pretty
+different set of options availabe as well, please see the API documentation for both sides.
+Note that, to match behaviours for PySpark to Pandas, `header=True` and `inferSchema=True`
+options are required.
+
+
+### Save DataFrame
+
+To load DataFrame, likewise, the APIs are `spark.read.xxx` (or `spark.read.format("xxx")`) for
+PySpark and `pandas_to_xxx` for Pandas.
+
+```
+df.to_csv("data.csv")
+```
+
+```
+df.write.csv("data.set")
+```
+
+Likewise, different set of sources are supported. Note that, to match behaviours for PySpark
+to Pandas, `header=True` option is required.
+
+
+### Inspect DataFrame
+
+DataFrame in both PySpark and Pandas can be expected in many ways. One of the way is to
+`describe()` as below:
+
+```
+# PySpark
+df.describe().show()
+```
+
+```
++-------+-------------+----+
+|summary| city|rank|
++-------+-------------+----+
+| count| 4| 4|
+| mean| null| NaN|
+| stddev| null| NaN|
+| min| Chicago| 1.0|
+| max|San Francisco| NaN|
++-------+-------------+----+
+```
+
+In Pandas, the same name function `describe()` can be called:
+
+```
+# Pandas
+df.describe()
+```
+
+```
+ rank
+count 3.0
+mean 2.0
+std 1.0
+min 1.0
+25% 1.5
+50% 2.0
+75% 2.5
+max 3.0
+```
+
+There are some differences, for instance, in case of PySpark, `NaN` is excluded but Pandas ignore `NaN` in some statistics.
+
+To inspect columns, there are `dtype` API in PySpark and Pandas.
+
+```
+# PySpark
+df.dtypes
+```
+
+```
+[('city', 'string'), ('rank', 'double')]
+```
+
+```
+# Pandas
+df.dtypes
+```
+
+```
+0 object
+1 float64
+dtype: object
+```
+
+PySpark types are represented as Spark SQL types whereas they are NumPy types in case of Pandas.
+
+Note that, PySpark supports pretty printing of schema as below:
+
+```
+# PySpark
+df.printSchema()
+```
+
+```
+root
+ |-- city: string (nullable = true)
+ |-- rank: double (nullable = true)
+```
+
+
+### Interaction between PySpark and Pandas
+
+PySpark supports conversion to/from Pandas and Pandas UDFs out of the box.
+See [PySpark Usage Guide for Pandas with Apache Arrow](https://spark.apache.org/docs/latest/sql-pyspark-pandas-with-arrow.html).
+
+
+
+## Notable Differences
+
+There are other notable differences for many reasons, for instance,
+PySpark DataFrame's core is Spark SQL so some of APIs comply SQL, and this might be different
+comparing to Pandas context which somtimes causes some subtleties. See the differences below.
+
+
+### Lazy and Eager Evaluation
+
+PySpark DataFrame is lazy whereas Pandas DataFrame is eager. Therefore,
+PySpark DataFrame can optimize the computation before an actual execution whereas Pandas
+DataFrame can have each direct result immediately after each execution.
+
+For sintance, suppose you select columns multiple times as below:
Review comment:
sintance -> instance
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org