You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafu.apache.org by ey...@apache.org on 2024/12/23 11:24:00 UTC
(datafu) branch main updated: Fix pyspark docs (#48)
This is an automated email from the ASF dual-hosted git repository.
eyal pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/datafu.git
The following commit(s) were added to refs/heads/main by this push:
new c962d5e Fix pyspark docs (#48)
c962d5e is described below
commit c962d5e000cf548815ed2e81788d2314b24f392d
Author: Ohad Raviv <15...@users.noreply.github.com>
AuthorDate: Mon Dec 23 13:23:55 2024 +0200
Fix pyspark docs (#48)
* Fix pyspark docs
* revert blog
* update readme version
---------
Co-authored-by: ohad <oh...@definity.ai>
---
datafu-spark/README.md | 19 +++++++------------
site/source/docs/spark/guide.html.markdown.erb | 19 ++++++++++---------
2 files changed, 17 insertions(+), 21 deletions(-)
diff --git a/datafu-spark/README.md b/datafu-spark/README.md
index f670067..152677b 100644
--- a/datafu-spark/README.md
+++ b/datafu-spark/README.md
@@ -34,19 +34,17 @@ In order to call the datafu-spark API's from Pyspark, you can do the following (
First, call pyspark with the following parameters
```bash
-export PYTHONPATH=datafu-spark_2.11-1.8.0.jar
+export PYTHONPATH=datafu-spark_2.12-2.0.0.jar
-pyspark --jars datafu-spark_2.11-1.8.0.jar --conf spark.executorEnv.PYTHONPATH=datafu-spark_2.11-1.8.0.jar
+pyspark --jars datafu-spark_2.12-2.0.0.jar --conf spark.executorEnv.PYTHONPATH=datafu-spark_2.12-2.0.0.jar
```
The following is an example of calling the Spark version of the datafu _dedup_ method
```python
-from pyspark_utils.df_utils import PySparkDFUtils
+from pyspark_utils import df_utils
-df_utils = PySparkDFUtils()
-
-df_people = sqlContext.createDataFrame([
+df_people = spark.createDataFrame([
("a", "Alice", 34),
("a", "Sara", 33),
("b", "Bob", 36),
@@ -57,12 +55,9 @@ df_people = sqlContext.createDataFrame([
("c", "Zoey", 36)],
["id", "name", "age"])
-func_dedup_res = df_utils.dedup_with_order(dataFrame=df_people, groupCol=df_people.id,
- orderCols=[df_people.age.desc(), df_people.name.desc()])
-
-func_dedup_res.registerTempTable("dedup")
-
-func_dedup_res.show()
+df_dedup = df_utils.dedup_with_order(df=df_people, group_col=df_people.id,
+ order_cols=[df_people.age.desc(), df_people.name.desc()])
+df_dedup.show()
```
This should produce the following output
diff --git a/site/source/docs/spark/guide.html.markdown.erb b/site/source/docs/spark/guide.html.markdown.erb
index 342b669..b373eab 100644
--- a/site/source/docs/spark/guide.html.markdown.erb
+++ b/site/source/docs/spark/guide.html.markdown.erb
@@ -43,11 +43,9 @@ pyspark --jars datafu-spark_2.12-<%= current_page.data.version %>-SNAPSHOT.jar -
The following is an example of calling the Spark version of the datafu _dedup_ method
```python
-from pyspark_utils.df_utils import PySparkDFUtils
+from pyspark_utils import df_utils
-df_utils = PySparkDFUtils()
-
-df_people = sqlContext.createDataFrame([
+df_people = spark.createDataFrame([
("a", "Alice", 34),
("a", "Sara", 33),
("b", "Bob", 36),
@@ -58,12 +56,15 @@ df_people = sqlContext.createDataFrame([
("c", "Zoey", 36)],
["id", "name", "age"])
-func_dedup_res = df_utils.dedup_with_order(dataFrame=df_people, groupCol=df_people.id,
- orderCols=[df_people.age.desc(), df_people.name.desc()])
-
-func_dedup_res.registerTempTable("dedup")
+df_dedup = df_utils.dedup_with_order(df=df_people, group_col=df_people.id,
+ order_cols=[df_people.age.desc(), df_people.name.desc()])
+df_dedup.show()
-func_dedup_res.show()
+# or with activate()
+df_utils.activate()
+df_dedup_top_n = df_people.dedup_top_n(n=2, group_col=df_people.id,
+ order_cols=[df_people.age.desc(), df_people.name.desc()])
+df_dedup_top_n.show()
```
This should produce the following output