You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by "zhengruifeng (via GitHub)" <gi...@apache.org> on 2023/12/13 09:25:23 UTC

[PR] [SPARK-46390] Update the `Example` page [spark-website]

zhengruifeng opened a new pull request, #493:
URL: https://github.com/apache/spark-website/pull/493

   1, remove the out-dated `RDD API examples`;
   2, update `DataFrame API examples` and `Machine learning example`:
   
   - `sc.textFile(...).map(...).toDF` -> `spark.read.text`
   - `sqlContext.read` -> `spark.read`
   - `sqlContext.createDataFrame` -> `spark.createDataFrame`
   
   
   ![image](https://github.com/apache/spark-website/assets/7322292/c2a62d76-6043-41d7-bed1-8460e3c0b6dc)
   
   ![image](https://github.com/apache/spark-website/assets/7322292/8f601785-34f7-43a2-9e67-51343e08518a)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Re: [PR] [SPARK-46390] Update the `Example` page [spark-website]

Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #493:
URL: https://github.com/apache/spark-website/pull/493#issuecomment-1857789907

   @viirya @srowen @HyukjinKwon thank you so much for the reviews.
   
   FYI, @MrPowers is preparing a new version of this page with much better examples, I am going to close this PR in favor of the new one


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Re: [PR] [SPARK-46390] Update the `Example` page [spark-website]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #493:
URL: https://github.com/apache/spark-website/pull/493#discussion_r1426318473


##########
examples.md:
##########
@@ -36,36 +42,38 @@ In this page, we will show examples using RDD API as well as examples using high
 <div class="tab-pane tab-pane-python active">
 <div class="code code-tab">
 {% highlight python %}
-text_file = sc.textFile("hdfs://...")
-counts = text_file.flatMap(lambda line: line.split(" ")) \
-             .map(lambda word: (word, 1)) \
-             .reduceByKey(lambda a, b: a + b)
-counts.saveAsTextFile("hdfs://...")
+df = spark.read.text("hdfs://...").toDF("text")
+
+df.select(explode(split(col("text"), " ")).alias("word")) \
+  .groupBy("word") \
+  .agg(count(lit(0)).alias("count")) \
+  .write.parquet("hdfs://...")
 {% endhighlight %}
 </div>
 </div>
 
 <div class="tab-pane tab-pane-scala">
 <div class="code code-tab">
 {% highlight scala %}
-val textFile = sc.textFile("hdfs://...")
-val counts = textFile.flatMap(line => line.split(" "))
-                 .map(word => (word, 1))
-                 .reduceByKey(_ + _)
-counts.saveAsTextFile("hdfs://...")
+val df = spark.read.text("hdfs://...").toDF("text")
+
+df.select(explode(split(col("text"), " ")).alias("word"))
+  .groupBy("word")
+  .agg(count(lit(0)).alias("count"))
+  .write.parquet("hdfs://...")
 {% endhighlight %}
 </div>
 </div>
 
 <div class="tab-pane tab-pane-java">
 <div class="code code-tab">
 {% highlight java %}
-JavaRDD<String> textFile = sc.textFile("hdfs://...");
-JavaPairRDD<String, Integer> counts = textFile
-    .flatMap(s -> Arrays.asList(s.split(" ")).iterator())
-    .mapToPair(word -> new Tuple2<>(word, 1))
-    .reduceByKey((a, b) -> a + b);
-counts.saveAsTextFile("hdfs://...");
+DataFrame df = spark.read.text("hdfs://...").toDF("text");

Review Comment:
   Hmm, for Java, isn't it `read()`?
   ```suggestion
   DataFrame df = spark.read().text("hdfs://...").toDF("text");
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Re: [PR] [SPARK-46390] Update the `Example` page [spark-website]

Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng closed pull request #493: [SPARK-46390] Update the `Example` page
URL: https://github.com/apache/spark-website/pull/493


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Re: [PR] [SPARK-46390] Update the `Example` page [spark-website]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #493:
URL: https://github.com/apache/spark-website/pull/493#discussion_r1426302479


##########
examples.md:
##########
@@ -19,12 +19,18 @@ On top of Spark’s RDD API, high level APIs are provided, e.g.
 [DataFrame API](https://spark.apache.org/docs/latest/sql-programming-guide.html#datasets-and-dataframes) and
 [Machine Learning API](https://spark.apache.org/docs/latest/mllib-guide.html).
 These high level APIs provide a concise way to conduct certain data operations.
-In this page, we will show examples using RDD API as well as examples using high level APIs.
 
-<h2>RDD API examples</h2>
+<h2>DataFrame API examples</h2>
+<p>
+In Spark, a <a href="https://spark.apache.org/docs/latest/sql-programming-guide.html#dataframes">DataFrame</a>
+is a distributed collection of data organized into named columns.
+Users can use DataFrame API to perform various relational operations on both external
+data sources and Spark’s built-in distributed collections without providing specific procedures for processing data.
+Also, programs based on DataFrame API will be automatically optimized by Spark’s built-in optimizer, Catalyst.
+</p>
 
 <h3>Word count</h3>
-<p>In this example, we use a few transformations to build a dataset of (String, Int) pairs called <code>counts</code> and then save it to a file.</p>
+<p>In this example, we use a few transformations to build a dataset of (String, Long) pairs and then save it to a file.</p>

Review Comment:
   Hmm, I think `transformations` are terms used for RDD operations (i.e, `transformations` and `actions` as this page mentioned above)? For DataFrame, maybe `relational operations`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Re: [PR] [SPARK-46390] Update the `Example` page [spark-website]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #493:
URL: https://github.com/apache/spark-website/pull/493#discussion_r1426326048


##########
examples.md:
##########
@@ -36,36 +42,38 @@ In this page, we will show examples using RDD API as well as examples using high
 <div class="tab-pane tab-pane-python active">
 <div class="code code-tab">
 {% highlight python %}
-text_file = sc.textFile("hdfs://...")
-counts = text_file.flatMap(lambda line: line.split(" ")) \
-             .map(lambda word: (word, 1)) \
-             .reduceByKey(lambda a, b: a + b)
-counts.saveAsTextFile("hdfs://...")
+df = spark.read.text("hdfs://...").toDF("text")
+
+df.select(explode(split(col("text"), " ")).alias("word")) \
+  .groupBy("word") \
+  .agg(count(lit(0)).alias("count")) \
+  .write.parquet("hdfs://...")
 {% endhighlight %}
 </div>
 </div>
 
 <div class="tab-pane tab-pane-scala">
 <div class="code code-tab">
 {% highlight scala %}
-val textFile = sc.textFile("hdfs://...")
-val counts = textFile.flatMap(line => line.split(" "))
-                 .map(word => (word, 1))
-                 .reduceByKey(_ + _)
-counts.saveAsTextFile("hdfs://...")
+val df = spark.read.text("hdfs://...").toDF("text")
+
+df.select(explode(split(col("text"), " ")).alias("word"))
+  .groupBy("word")
+  .agg(count(lit(0)).alias("count"))
+  .write.parquet("hdfs://...")
 {% endhighlight %}
 </div>
 </div>
 
 <div class="tab-pane tab-pane-java">
 <div class="code code-tab">
 {% highlight java %}
-JavaRDD<String> textFile = sc.textFile("hdfs://...");
-JavaPairRDD<String, Integer> counts = textFile
-    .flatMap(s -> Arrays.asList(s.split(" ")).iterator())
-    .mapToPair(word -> new Tuple2<>(word, 1))
-    .reduceByKey((a, b) -> a + b);
-counts.saveAsTextFile("hdfs://...");
+DataFrame df = spark.read.text("hdfs://...").toDF("text");
+
+df.select(explode(split(col("text"), " ")).alias("word"))
+  .groupBy("word")
+  .agg(count(lit(0)).alias("count"))
+  .write.parquet("hdfs://...");

Review Comment:
   ```suggestion
   df.select(explode(split(col("text"), " ")).alias("word"))
     .groupBy("word")
     .agg(count(lit(0)).alias("count"))
     .write().parquet("hdfs://...");
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Re: [PR] [SPARK-46390] Update the `Example` page [spark-website]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #493:
URL: https://github.com/apache/spark-website/pull/493#discussion_r1426323076


##########
examples.md:
##########
@@ -186,13 +178,7 @@ errors.filter(col("line").like("%MySQL%")).collect()
 <div class="code code-tab">
 {% highlight java %}
 // Creates a DataFrame having a single column named "line"
-JavaRDD<String> textFile = sc.textFile("hdfs://...");
-JavaRDD<Row> rowRDD = textFile.map(RowFactory::create);
-List<StructField> fields = Arrays.asList(
-  DataTypes.createStructField("line", DataTypes.StringType, true));
-StructType schema = DataTypes.createStructType(fields);
-DataFrame df = sqlContext.createDataFrame(rowRDD, schema);
-
+DataFrame df = spark.read.text("hdfs://...").toDF("line");

Review Comment:
   ```suggestion
   DataFrame df = spark.read().text("hdfs://...").toDF("line");
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Re: [PR] [SPARK-46390] Update the `Example` page [spark-website]

Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #493:
URL: https://github.com/apache/spark-website/pull/493#issuecomment-1854923264

   I want to make a few more modification, let me convert it to draft first


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org