You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by Mironor <gi...@git.apache.org> on 2016/12/04 18:35:28 UTC
[GitHub] spark pull request #16137: [SPARK-18708][CORE] Improvement/improve docs in s...
GitHub user Mironor opened a pull request:
https://github.com/apache/spark/pull/16137
[SPARK-18708][CORE] Improvement/improve docs in spark context file
## What changes were proposed in this pull request?
SparkContext.scala was created a long time ago and contains several types of Scaladocs/Javadocs mixed together. Public methods/fields should have a Scaladoc that is formatted in the same way everywhere. This pull request also adds scaladoc to methods/fields that did not have it before.
## How was this patch tested?
No actual code was modified, only comments.
Please review http://spark.apache.org/contributing.html before opening a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/Mironor/spark improvement/improve-docs-in-spark-context-file
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/16137.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #16137
----
commit 87bc74fb05a665b3972ce56e96df66dfd4c457fd
Author: Aliaksandr.Bedrytski <al...@valtech.co.uk>
Date: 2016-12-04T18:24:43Z
Improves code style of the documentation in SparkContext.scala
Adds documentation to public methods/fields that do not have it
commit 5d5236baab7c7889b9153fcd9c27b4839f075a8f
Author: Aliaksandr.Bedrytski <al...@valtech.co.uk>
Date: 2016-12-04T18:30:26Z
Modifies line that exceeds 100 characters
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16137: [SPARK-18708][CORE] Improvement/improve docs in s...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/16137#discussion_r91009699
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1620,8 +1766,9 @@ class SparkContext(config: SparkConf) extends Logging {
/**
* :: DeveloperApi ::
- * Return information about what RDDs are cached, if they are in mem or on disk, how much space
- * they take, etc.
--- End diff --
If you want to remove `Return` or make the description into `@returns`, I guess it should be at least consistent. It seem https://github.com/apache/spark/pull/16137/files#r91009309 is a bit different.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16137: [SPARK-18708][CORE] Improvement/improve docs in spark co...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/16137
Up to you but it's fine to just start from scratch in a new PR. Sometimes that's simpler, esp. because it will probably be fairly different anyway.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16137: [SPARK-18708][CORE] Improvement/improve docs in s...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/16137#discussion_r91855194
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -721,16 +732,15 @@ class SparkContext(config: SparkConf) extends Logging {
}
/**
- * Creates a new RDD[Long] containing elements from `start` to `end`(exclusive), increased by
+ * Creates a new `RDD[Long]` containing elements from `start` to `end`(exclusive), increased by
* `step` every element.
*
* @note if we need to cache this RDD, we should make sure each partition does not exceed limit.
- *
* @param start the start value.
* @param end the end value.
* @param step the incremental step
* @param numSlices the partition number of the new RDD.
- * @return
+ * @return RDD containing distributed range
--- End diff --
Nit: RDDs are really pointers and bookeeping. The "represent" rather than "contain"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16137: [SPARK-18708][CORE] Improvement/improve docs in spark co...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/16137
Yes I'm referring to
https://github.com/apache/spark/pull/16137#discussion_r90948858
and to some extent these issues remain in some form, though it's a judgment call:
https://github.com/apache/spark/pull/16137#discussion_r90948903
https://github.com/apache/spark/pull/16137#discussion_r91170210
Given the sense of the feedback I think you can probably extrapolate what of this is clearly useful and what probably isn't worth it. If you can take your best shot at applying that to the whole change then I'll have another look.
No, don't split up the PR. I just meant that it wouldn't be useful to make 20 variations of the same comments if you can just run with them and apply to the rest first.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16137: [SPARK-18708][CORE] Improvement/improve docs in s...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/16137#discussion_r91010366
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1417,27 +1551,31 @@ class SparkContext(config: SparkConf) extends Logging {
/**
* Add a file to be downloaded with this Spark job on every node.
- * The `path` passed can be either a local file, a file in HDFS (or other Hadoop-supported
- * filesystems), or an HTTP, HTTPS or FTP URI. To access the file in Spark jobs,
- * use `SparkFiles.get(fileName)` to find its download location.
+ *
+ * @param path can be either a local file, a file in HDFS (or other Hadoop-supported
+ * filesystems), or an HTTP, HTTPS or FTP URI. To access the file in Spark jobs,
+ * use `SparkFiles.get(fileName)` to find its download location.
*/
def addFile(path: String): Unit = {
addFile(path, false)
}
/**
- * Returns a list of file paths that are added to resources.
+ * A list of file paths that are added to resources.
--- End diff --
We shouldn't duplicate the documentation. converting this to a `@return` is fine.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16137: [SPARK-18708][CORE] Improvement/improve docs in s...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/16137#discussion_r90948705
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -119,22 +119,22 @@ class SparkContext(config: SparkConf) extends Logging {
/**
* Alternative constructor that allows setting common Spark properties directly
*
- * @param master Cluster URL to connect to (e.g. mesos://host:port, spark://host:port, local[4]).
- * @param appName A name for your application, to display on the cluster web UI
- * @param conf a [[org.apache.spark.SparkConf]] object specifying other Spark parameters
+ * @param master cluster URL to connect to (e.g. mesos://host:port, spark://host:port, local[4]).
+ * @param appName a name for your application, to display on the cluster web UI
+ * @param conf a `org.apache.spark.SparkConf` object specifying other Spark parameters
--- End diff --
Does it not render properly? then just `SparkConf` is fine
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16137: [SPARK-18708][CORE] Improvement/improve docs in s...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/16137#discussion_r91169186
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -358,9 +361,11 @@ class SparkContext(config: SparkConf) extends Logging {
}
}
- /** Control our logLevel. This overrides any user-defined log settings.
+ /**
+ * Control our logLevel. This overrides any user-defined log settings.
+ *
* @param logLevel The desired log level as a string.
- * Valid log levels include: ALL, DEBUG, ERROR, FATAL, INFO, OFF, TRACE, WARN
+ * Valid log levels include: ALL, DEBUG, ERROR, FATAL, INFO, OFF, TRACE, WARN
--- End diff --
Yes we don't need continuation indents. The code doesn't use it in general and wastes some space
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16137: [SPARK-18708][CORE] Improvement/improve docs in s...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/16137#discussion_r91008183
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1417,27 +1551,31 @@ class SparkContext(config: SparkConf) extends Logging {
/**
* Add a file to be downloaded with this Spark job on every node.
- * The `path` passed can be either a local file, a file in HDFS (or other Hadoop-supported
- * filesystems), or an HTTP, HTTPS or FTP URI. To access the file in Spark jobs,
- * use `SparkFiles.get(fileName)` to find its download location.
+ *
+ * @param path can be either a local file, a file in HDFS (or other Hadoop-supported
+ * filesystems), or an HTTP, HTTPS or FTP URI. To access the file in Spark jobs,
--- End diff --
nit: we could make this single spaced here` URI. To` and also same instances, at least for the lines this PR changes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16137: [SPARK-18708][CORE] Improvement/improve docs in spark co...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/16137
Ping @Mironor are you able to pare this down further? I want to figure out whether this will proceed or not. It needs a rebase anyhow.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16137: [SPARK-18708][CORE] Improvement/improve docs in s...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/16137#discussion_r91938097
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -2307,6 +2432,8 @@ object SparkContext extends Logging {
*
* @note This function cannot be used to create multiple SparkContext instances
* even if multiple contexts are allowed.
+ * @param config Spark's config that will be used for initialisation of the `SparkContext`
--- End diff --
`SparkConf` instead of `Spark's config` ? just because elsewhere you're referring to concepts by class name in this line.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16137: [SPARK-18708][CORE] Improvement/improve docs in s...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/16137#discussion_r91855251
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -956,24 +976,24 @@ class SparkContext(config: SparkConf) extends Logging {
}
/**
- * Get an RDD for a Hadoop-readable dataset from a Hadoop JobConf given its InputFormat and other
- * necessary info (e.g. file name for a filesystem-based dataset, table name for HyperTable),
- * using the older MapReduce API (`org.apache.hadoop.mapred`).
+ * Get an RDD for a Hadoop-readable dataset from a Hadoop `JobConf` given its `InputFormat`
+ * and other necessary info (e.g. file name for a filesystem-based dataset, table name
+ * for HyperTable), using the older MapReduce API (`org.apache.hadoop.mapred`).
*
- * @param conf JobConf for setting up the dataset. Note: This will be put into a Broadcast.
+ * @note Because Hadoop's `RecordReader` class re-uses the same Writable object for each
+ * record, directly caching the returned RDD or directly passing it to an aggregation
+ * or shuffle operation will create many references to the same object.
+ * If you plan to directly cache, sort, or aggregate Hadoop writable objects, you
+ * should first copy them using a `map` function.
+ * @param conf `JobConf` for setting up the dataset. Note: This will be put into a Broadcast.
* Therefore if you plan to reuse this conf to create multiple RDDs, you need to make
* sure you won't modify the conf. A safe approach is always creating a new conf for
* a new RDD.
* @param inputFormatClass Class of the InputFormat
* @param keyClass Class of the keys
* @param valueClass Class of the values
- * @param minPartitions Minimum number of Hadoop Splits to generate.
- *
- * @note Because Hadoop's RecordReader class re-uses the same Writable object for each
- * record, directly caching the returned RDD or directly passing it to an aggregation or shuffle
- * operation will create many references to the same object.
- * If you plan to directly cache, sort, or aggregate Hadoop writable objects, you should first
- * copy them using a `map` function.
+ * @param minPartitions minimum number of Hadoop Splits to generate.
--- End diff --
Say "partitions" not "splits" despite what the existing string says
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16137: [SPARK-18708][CORE] Improvement/improve docs in spark co...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16137
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16137: [SPARK-18708][CORE] Improvement/improve docs in s...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/16137#discussion_r90948641
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -119,22 +119,22 @@ class SparkContext(config: SparkConf) extends Logging {
/**
* Alternative constructor that allows setting common Spark properties directly
*
- * @param master Cluster URL to connect to (e.g. mesos://host:port, spark://host:port, local[4]).
- * @param appName A name for your application, to display on the cluster web UI
- * @param conf a [[org.apache.spark.SparkConf]] object specifying other Spark parameters
+ * @param master cluster URL to connect to (e.g. mesos://host:port, spark://host:port, local[4]).
--- End diff --
It's not worth un-capitalizing these words
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16137: [SPARK-18708][CORE] Improvement/improve docs in spark co...
Posted by Mironor <gi...@git.apache.org>.
Github user Mironor commented on the issue:
https://github.com/apache/spark/pull/16137
I removed all trivial (unhelpful) additions and expended javadoc where it didn't add any value to the method signature. I payed special attention to all your previous comments and reflected them in the whole file.
The actual diff was reduced from the initial 400+ lines to 150. This is my best shot at this PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16137: [SPARK-18708][CORE] Improvement/improve docs in s...
Posted by Mironor <gi...@git.apache.org>.
Github user Mironor commented on a diff in the pull request:
https://github.com/apache/spark/pull/16137#discussion_r91906441
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1949,20 +2038,35 @@ class SparkContext(config: SparkConf) extends Logging {
/**
* Run a job on all partitions in an RDD and return the results in an array.
+ *
+ * @param rdd target RDD to run tasks on
+ * @param func a function to run on each partition of the RDD
+ * @return in-memory collection with a result of the job (each collection element will contain
+ * a result from one partition)
*/
def runJob[T, U: ClassTag](rdd: RDD[T], func: (TaskContext, Iterator[T]) => U): Array[U] = {
runJob(rdd, func, 0 until rdd.partitions.length)
}
/**
- * Run a job on all partitions in an RDD and return the results in an array.
+ * Run a job on all partitions in an RDD and return the results in an array, but take a function
+ * of type `Iterator[T] => U` instead of `(TaskContext, Iterator[T]) => U`.
--- End diff --
Two `runJob` methods had exactly the same javadoc, I added this line (similar to the already existing line 2020) to show the difference between them.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16137: [SPARK-18708][CORE] Improvement/improve docs in s...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/16137#discussion_r91010001
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1401,8 +1532,11 @@ class SparkContext(config: SparkConf) extends Logging {
/**
* Broadcast a read-only variable to the cluster, returning a
- * [[org.apache.spark.broadcast.Broadcast]] object for reading it in distributed functions.
+ * `org.apache.spark.broadcast.Broadcast` object for reading it in distributed functions.
--- End diff --
Actually brackets are better because they make links. Some were backquoted because javadoc8 complains about this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16137: [SPARK-18708][CORE] Improvement/improve docs in spark co...
Posted by Mironor <gi...@git.apache.org>.
Github user Mironor commented on the issue:
https://github.com/apache/spark/pull/16137
Right, I see what you are referring to. I will apply those ones to a greater degree than before.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16137: [SPARK-18708][CORE] Improvement/improve docs in s...
Posted by Mironor <gi...@git.apache.org>.
Github user Mironor commented on a diff in the pull request:
https://github.com/apache/spark/pull/16137#discussion_r91856040
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -956,24 +976,24 @@ class SparkContext(config: SparkConf) extends Logging {
}
/**
- * Get an RDD for a Hadoop-readable dataset from a Hadoop JobConf given its InputFormat and other
- * necessary info (e.g. file name for a filesystem-based dataset, table name for HyperTable),
- * using the older MapReduce API (`org.apache.hadoop.mapred`).
+ * Get an RDD for a Hadoop-readable dataset from a Hadoop `JobConf` given its `InputFormat`
+ * and other necessary info (e.g. file name for a filesystem-based dataset, table name
+ * for HyperTable), using the older MapReduce API (`org.apache.hadoop.mapred`).
*
- * @param conf JobConf for setting up the dataset. Note: This will be put into a Broadcast.
+ * @note Because Hadoop's `RecordReader` class re-uses the same Writable object for each
+ * record, directly caching the returned RDD or directly passing it to an aggregation
+ * or shuffle operation will create many references to the same object.
+ * If you plan to directly cache, sort, or aggregate Hadoop writable objects, you
+ * should first copy them using a `map` function.
+ * @param conf `JobConf` for setting up the dataset. Note: This will be put into a Broadcast.
* Therefore if you plan to reuse this conf to create multiple RDDs, you need to make
* sure you won't modify the conf. A safe approach is always creating a new conf for
* a new RDD.
* @param inputFormatClass Class of the InputFormat
* @param keyClass Class of the keys
* @param valueClass Class of the values
- * @param minPartitions Minimum number of Hadoop Splits to generate.
- *
--- End diff --
This was my answer to your original comment about moving this:
> Following 2 functions have @note above the parameter description, that was the reason why I moved it. Its content is directly related to the returned value, so it may be justified to move it after the @return statement, but in this case the doc of the following 2 functions could be modified as well.
As I understand, you mean that this is not worth moving because of the possible merge conflicts, right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16137: [SPARK-18708][CORE] Improvement/improve docs in s...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/16137#discussion_r91855189
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -705,13 +712,17 @@ class SparkContext(config: SparkConf) extends Logging {
// Methods for creating RDDs
- /** Distribute a local Scala collection to form an RDD.
+ /**
+ * Distribute a local Scala collection to form an RDD.
*
* @note Parallelize acts lazily. If `seq` is a mutable collection and is altered after the call
* to parallelize and before the first action on the RDD, the resultant RDD will reflect the
* modified collection. Pass a copy of the argument to avoid this.
* @note avoid using `parallelize(Seq())` to create an empty `RDD`. Consider `emptyRDD` for an
* RDD with no partitions, or `parallelize(Seq[T]())` for an RDD of `T` with empty partitions.
+ * @param seq Scala collection to distribute
+ * @param numSlices how many sub-collections should it be sliced into
--- End diff --
numSlices is a legacy name. Although the name must be kept its doc should refer to "partitions". How about "number of partitions to divide collection into"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16137: [SPARK-18708][CORE] Improvement/improve docs in s...
Posted by Mironor <gi...@git.apache.org>.
Github user Mironor commented on a diff in the pull request:
https://github.com/apache/spark/pull/16137#discussion_r91009631
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1417,27 +1551,31 @@ class SparkContext(config: SparkConf) extends Logging {
/**
* Add a file to be downloaded with this Spark job on every node.
- * The `path` passed can be either a local file, a file in HDFS (or other Hadoop-supported
- * filesystems), or an HTTP, HTTPS or FTP URI. To access the file in Spark jobs,
- * use `SparkFiles.get(fileName)` to find its download location.
+ *
+ * @param path can be either a local file, a file in HDFS (or other Hadoop-supported
+ * filesystems), or an HTTP, HTTPS or FTP URI. To access the file in Spark jobs,
+ * use `SparkFiles.get(fileName)` to find its download location.
*/
def addFile(path: String): Unit = {
addFile(path, false)
}
/**
- * Returns a list of file paths that are added to resources.
+ * A list of file paths that are added to resources.
--- End diff --
The second line is redundant here, my question here is whether it's worth to replace `Return` with `@return` or just to leave it as it is.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16137: [SPARK-18708][CORE] Improvement/improve docs in spark co...
Posted by Mironor <gi...@git.apache.org>.
Github user Mironor commented on the issue:
https://github.com/apache/spark/pull/16137
I addressed the feedback about the things that are left to modify. Thank you for taking your time to review this PR, I learnt a lot in terms of your expectations of what is useful and what isn't in a pull request.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16137: [SPARK-18708][CORE] Improvement/improve docs in spark co...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/16137
Merged to master
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16137: [SPARK-18708][CORE] Improvement/improve docs in s...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/16137#discussion_r91934287
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1038,6 +1059,13 @@ class SparkContext(config: SparkConf) extends Logging {
* operation will create many references to the same object.
* If you plan to directly cache, sort, or aggregate Hadoop writable objects, you should first
* copy them using a `map` function.
+ * @param path directory to the input data files, the path can be comma separated paths
+ * as a list of inputs
+ * @param minPartitions suggested minimum number of partitions for the resulting RDD
+ * @param km implicitly found Class of the key associated with the `fm` implicit parameter
--- End diff --
Interesting, are these javadoc-able? a caller wouldn't really specify them explicitly. Does javadoc succeed when you build with this change? I don't know if these docs help much; they just say it's some implicit thing associated to another value. I wouldn't add docs just for completeness.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16137: [SPARK-18708][CORE] Improvement/improve docs in spark co...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/16137
It is good to start small, but not all small changes are worthwhile. It would be better to focus on fixing actual problems like a typo or missing doc. You can then expand to improve / fix docs across many files, not just one.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16137: [SPARK-18708][CORE] Improvement/improve docs in s...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/16137#discussion_r91855172
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -486,9 +491,11 @@ class SparkContext(config: SparkConf) extends Logging {
value <- Option(System.getenv(envKey)).orElse(Option(System.getProperty(propKey)))} {
executorEnvs(envKey) = value
}
+
--- End diff --
Likewise whitespace changes just aren't important and have a tiny cost (merge conflicts)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16137: [SPARK-18708][CORE] Improvement/improve docs in s...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/16137#discussion_r91015460
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1144,13 +1218,19 @@ class SparkContext(config: SparkConf) extends Logging {
}
/**
- * Get an RDD for a Hadoop SequenceFile with given key and value types.
+ * Get an RDD for a Hadoop `SequenceFile` with given key and value types.
*
- * @note Because Hadoop's RecordReader class re-uses the same Writable object for each
- * record, directly caching the returned RDD or directly passing it to an aggregation or shuffle
- * operation will create many references to the same object.
- * If you plan to directly cache, sort, or aggregate Hadoop writable objects, you should first
- * copy them using a `map` function.
+ * @note because Hadoop's `RecordReader` class re-uses the same `Writable` object for each
--- End diff --
I understand they are used in a mixed way and I see the example of multiple lines with `@return` in the scaladoc. I am fine with this but I just wanted to note my worry here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16137: [SPARK-18708][CORE] Improvement/improve docs in s...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/16137#discussion_r90948858
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -276,13 +279,17 @@ class SparkContext(config: SparkConf) extends Logging {
private[spark] def ui: Option[SparkUI] = _ui
+ /**
+ * @return `Option` containing url to web ui.
--- End diff --
url -> URL and ui -> UI, but, I don't this comment adds much to the reader beyond what the signature says already. I'd improve or remove it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16137: [SPARK-18708][CORE] Improvement/improve docs in s...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/16137#discussion_r91008727
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -119,22 +119,22 @@ class SparkContext(config: SparkConf) extends Logging {
/**
* Alternative constructor that allows setting common Spark properties directly
*
- * @param master Cluster URL to connect to (e.g. mesos://host:port, spark://host:port, local[4]).
- * @param appName A name for your application, to display on the cluster web UI
- * @param conf a [[org.apache.spark.SparkConf]] object specifying other Spark parameters
+ * @param master cluster URL to connect to (e.g. mesos://host:port, spark://host:port, local[4]).
+ * @param appName a name for your application, to display on the cluster web UI
+ * @param conf a `org.apache.spark.SparkConf` object specifying other Spark parameters
*/
def this(master: String, appName: String, conf: SparkConf) =
this(SparkContext.updatedConf(conf, master, appName))
/**
* Alternative constructor that allows setting common Spark properties directly
*
- * @param master Cluster URL to connect to (e.g. mesos://host:port, spark://host:port, local[4]).
- * @param appName A name for your application, to display on the cluster web UI.
- * @param sparkHome Location where Spark is installed on cluster nodes.
- * @param jars Collection of JARs to send to the cluster. These can be paths on the local file
+ * @param master cluster URL to connect to (e.g. mesos://host:port, spark://host:port, local[4]).
+ * @param appName a name for your application, to display on the cluster web UI.
+ * @param sparkHome location where Spark is installed on cluster nodes.
+ * @param jars collection of JARs to send to the cluster. These can be paths on the local file
* system or HDFS, HTTP, HTTPS, or FTP URLs.
- * @param environment Environment variables to set on worker nodes.
+ * @param environment environment variables to set on worker nodes.
--- End diff --
Oh, I am sorry it was mentioned in https://github.com/apache/spark/pull/16137/files#r90948641.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16137: [SPARK-18708][CORE] Improvement/improve docs in s...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/16137#discussion_r91937868
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1949,20 +2038,35 @@ class SparkContext(config: SparkConf) extends Logging {
/**
* Run a job on all partitions in an RDD and return the results in an array.
+ *
+ * @param rdd target RDD to run tasks on
+ * @param func a function to run on each partition of the RDD
+ * @return in-memory collection with a result of the job (each collection element will contain
+ * a result from one partition)
*/
def runJob[T, U: ClassTag](rdd: RDD[T], func: (TaskContext, Iterator[T]) => U): Array[U] = {
runJob(rdd, func, 0 until rdd.partitions.length)
}
/**
- * Run a job on all partitions in an RDD and return the results in an array.
+ * Run a job on all partitions in an RDD and return the results in an array, but take a function
+ * of type `Iterator[T] => U` instead of `(TaskContext, Iterator[T]) => U`.
--- End diff --
That's pretty OK, though the type info is already documented. The prose version of the difference is more like: this one's arg doesn't take a `TaskContext` and the other one does.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16137: [SPARK-18708][CORE] Improvement/improve docs in s...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/16137#discussion_r91170321
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -953,24 +977,24 @@ class SparkContext(config: SparkConf) extends Logging {
}
/**
- * Get an RDD for a Hadoop-readable dataset from a Hadoop JobConf given its InputFormat and other
- * necessary info (e.g. file name for a filesystem-based dataset, table name for HyperTable),
- * using the older MapReduce API (`org.apache.hadoop.mapred`).
+ * Get an RDD for a Hadoop-readable dataset from a Hadoop `JobConf` given its `InputFormat`
+ * and other necessary info (e.g. file name for a filesystem-based dataset, table name
+ * for HyperTable), using the older MapReduce API (`org.apache.hadoop.mapred`).
*
- * @param conf JobConf for setting up the dataset. Note: This will be put into a Broadcast.
+ * @note because Hadoop's `RecordReader` class re-uses the same Writable object for each
--- End diff --
Do you need to move this? you don't need to indent it
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16137: [SPARK-18708][CORE] Improvement/improve docs in s...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/16137#discussion_r90948903
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -276,13 +279,17 @@ class SparkContext(config: SparkConf) extends Logging {
private[spark] def ui: Option[SparkUI] = _ui
+ /**
+ * @return `Option` containing url to web ui.
+ */
def uiWebUrl: Option[String] = _ui.map(_.webUrl)
/**
* A default Hadoop Configuration for the Hadoop code (e.g. file systems) that we reuse.
*
- * @note As it will be reused in all Hadoop RDDs, it's better not to modify it unless you
- * plan to set some global configurations for all Hadoop RDDs.
+ * @note as it will be reused in all Hadoop RDDs, it's better not to modify it unless you
+ * plan to set some global configurations for all Hadoop RDDs.
+ * @return Hadoop configuration
--- End diff --
This isn't a useful comment; I'd revert changes like this
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16137: [SPARK-18708][CORE] Improvement/improve docs in s...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/16137#discussion_r90949013
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -461,11 +478,12 @@ class SparkContext(config: SparkConf) extends Logging {
_hadoopConfiguration = SparkHadoopUtil.get.newConfiguration(_conf)
- // Add each JAR given through the constructor
+ // Adds each JAR given through the constructor
--- End diff --
Ditto, lots of comments like this just aren't adding anything
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16137: [SPARK-18708][CORE] Improvement/improve docs in s...
Posted by Mironor <gi...@git.apache.org>.
Github user Mironor commented on a diff in the pull request:
https://github.com/apache/spark/pull/16137#discussion_r91015596
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1144,13 +1218,19 @@ class SparkContext(config: SparkConf) extends Logging {
}
/**
- * Get an RDD for a Hadoop SequenceFile with given key and value types.
+ * Get an RDD for a Hadoop `SequenceFile` with given key and value types.
*
- * @note Because Hadoop's RecordReader class re-uses the same Writable object for each
- * record, directly caching the returned RDD or directly passing it to an aggregation or shuffle
- * operation will create many references to the same object.
- * If you plan to directly cache, sort, or aggregate Hadoop writable objects, you should first
- * copy them using a `map` function.
+ * @note because Hadoop's `RecordReader` class re-uses the same `Writable` object for each
--- End diff --
Correct, but [they](http://www.oracle.com/technetwork/articles/java/index-137868.html) also contain continuation indentation (they even align parameter description)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16137: [SPARK-18708][CORE] Improvement/improve docs in s...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/16137#discussion_r91938033
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -2092,15 +2215,17 @@ class SparkContext(config: SparkConf) extends Logging {
* @param checkSerializable whether or not to immediately check <tt>f</tt> for serializability
* @throws SparkException if <tt>checkSerializable</tt> is set but <tt>f</tt> is not
* serializable
+ * @return reference cleaned from closures
--- End diff --
Might just say `@return the cleaned closure`; "cleaned from closures" doesn't really scan
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16137: [SPARK-18708][CORE] Improvement/improve docs in s...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/16137#discussion_r91170210
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -901,6 +926,7 @@ class SparkContext(config: SparkConf) extends Logging {
* @param path Directory to the input data files, the path can be comma separated paths as the
* list of inputs.
* @param minPartitions A suggestion value of the minimal splitting number for input data.
+ * @return RDD containing `Tuple2` with path to the file and the corresponding file content
--- End diff --
Although these don't add a lot it's kind of nice to doc `@return` for completeness here. I might change the language in all similar docs a bit to something more like "return an RDD of tuples of file path and corresponding file content". That is there are many tuples.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16137: [SPARK-18708][CORE] Improvement/improve docs in s...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/16137#discussion_r91169111
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -64,11 +64,11 @@ import org.apache.spark.util._
* Main entry point for Spark functionality. A SparkContext represents the connection to a Spark
* cluster, and can be used to create RDDs, accumulators and broadcast variables on that cluster.
*
- * Only one SparkContext may be active per JVM. You must `stop()` the active SparkContext before
- * creating a new one. This limitation may eventually be removed; see SPARK-2243 for more details.
+ * Only one SparkContext may be active per JVM. You must `stop()` the active SparkContext before
--- End diff --
I still just don't think it's worth removing spaces. They don't even matter in rendered HTML
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16137: [SPARK-18708][CORE] Improvement/improve docs in s...
Posted by Mironor <gi...@git.apache.org>.
Github user Mironor commented on a diff in the pull request:
https://github.com/apache/spark/pull/16137#discussion_r91013560
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1144,13 +1218,19 @@ class SparkContext(config: SparkConf) extends Logging {
}
/**
- * Get an RDD for a Hadoop SequenceFile with given key and value types.
+ * Get an RDD for a Hadoop `SequenceFile` with given key and value types.
*
- * @note Because Hadoop's RecordReader class re-uses the same Writable object for each
- * record, directly caching the returned RDD or directly passing it to an aggregation or shuffle
- * operation will create many references to the same object.
- * If you plan to directly cache, sort, or aggregate Hadoop writable objects, you should first
- * copy them using a `map` function.
+ * @note because Hadoop's `RecordReader` class re-uses the same `Writable` object for each
--- End diff --
[Spark style guide](http://spark.apache.org/contributing.html) doesn't contain anything about continuation indentation and refers to the [Scala's own style guide](http://docs.scala-lang.org/style/scaladoc.html) which shows that indentation should be used.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16137: [SPARK-18708][CORE] Improvement/improve docs in spark co...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16137
**[Test build #3504 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3504/testReport)** for PR 16137 at commit [`ffdca75`](https://github.com/apache/spark/commit/ffdca7546556452765dfbf6814c950782c9f7851).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16137: [SPARK-18708][CORE] Improvement/improve docs in s...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/16137#discussion_r91855158
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -276,6 +276,9 @@ class SparkContext(config: SparkConf) extends Logging {
private[spark] def ui: Option[SparkUI] = _ui
+ /**
+ * @return `Option` containing URL to web UI.
--- End diff --
This kind of doc doesn't add value, because it doesn't add anything the method signature doesn't already say. I'd remove it or expand it. It's picky, but I think we did assert this before.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16137: [SPARK-18708][CORE] Improvement/improve docs in spark co...
Posted by Mironor <gi...@git.apache.org>.
Github user Mironor commented on the issue:
https://github.com/apache/spark/pull/16137
@srowen I fixed everything. All comments are squashed, please review the current one.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16137: [SPARK-18708][CORE] Improvement/improve docs in spark co...
Posted by Mironor <gi...@git.apache.org>.
Github user Mironor commented on the issue:
https://github.com/apache/spark/pull/16137
@srowen
> I don't think most of my comment were addressed. This will be easier if you try to respond to each of them and apply them across your change at each round.
I genuinely would like to know which ones. I removed obvious javadocs (with the exception of `uiWebUrl` because I misunderstood you), indentation, upper-casing and applied suggested modifications to all the comments in this PR.
I also don't know if I should reply to everyone fo your comments or if it's just ok to apply the modification (so that the comment would be "collapsed" after the push).
> Otherwise, for a relatively minor change this takes a lot of bandwidth.
I could remove the difference below a certain line to make this PR more lightweight.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16137: [SPARK-18708][CORE] Improvement/improve docs in s...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/16137#discussion_r91169287
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -670,10 +677,14 @@ class SparkContext(config: SparkConf) extends Logging {
* sc.cancelJobGroup("some_job_to_cancel")
* }}}
*
- * If interruptOnCancel is set to true for the job group, then job cancellation will result
- * in Thread.interrupt() being called on the job's executor threads. This is useful to help ensure
- * that the tasks are actually stopped in a timely manner, but is off by default due to HDFS-1208,
- * where HDFS may respond to Thread.interrupt() by marking nodes as dead.
+ * @param groupId job group's identification
+ * @param description job group's decription
--- End diff --
The `interruptOnCancel` doc makes sense but these don't say much beyond the name. I'd expand them or remove them.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16137: [SPARK-18708][CORE] Improvement/improve docs in spark co...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16137
**[Test build #3504 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3504/testReport)** for PR 16137 at commit [`ffdca75`](https://github.com/apache/spark/commit/ffdca7546556452765dfbf6814c950782c9f7851).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16137: [SPARK-18708][CORE] Improvement/improve docs in s...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/16137#discussion_r91937528
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -816,6 +824,9 @@ class SparkContext(config: SparkConf) extends Logging {
/**
* Read a text file from HDFS, a local file system (available on all nodes), or any
* Hadoop-supported file system URI, and return it as an RDD of Strings.
+ * @param path path to the text file on a supported file system
+ * @param minPartitions suggested minimum number of partitions for the resulting RDD
+ * @return RDD representing the content of the file
--- End diff --
I still might lean towards saying more, if saying anything at all here: it's an RDD of lines of the text file. Many of these comments are OK but minimal.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16137: [SPARK-18708][CORE] Improvement/improve docs in s...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/16137#discussion_r91008467
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -923,15 +971,13 @@ class SparkContext(config: SparkConf) extends Logging {
/**
* Load data from a flat binary file, assuming the length of each record is constant.
*
- * @note We ensure that the byte array for each record in the resulting RDD
+ * @note we ensure that the byte array for each record in the resulting RDD
--- End diff --
I am not sure of making this lower-cased because we will see this as the start of sentence
- scaladoc

- javadoc

---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16137: [SPARK-18708][CORE] Improvement/improve docs in s...
Posted by Mironor <gi...@git.apache.org>.
Github user Mironor commented on a diff in the pull request:
https://github.com/apache/spark/pull/16137#discussion_r92595845
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -2067,8 +2067,8 @@ class SparkContext(config: SparkConf) extends Logging {
}
/**
- * Run a job on all partitions in an RDD and pass the results to a handler function, but take
- * a function of type `Iterator[T] => U` instead of `(TaskContext, Iterator[T]) => U`.
+ * Run a job on all partitions in an RDD and pass the results to a handler function, this time the
--- End diff --
If I understood you correctly, both of the functions should have `X` as the description, but the one with `TaskContext` has this added to its description: `The function that is run against each partition should additionally take TaskContext argument.`, right?
At the same time the actual argument description stays the same in both cases because the `TaskContext` thing is already described in the method signature.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16137: [SPARK-18708][CORE] Improvement/improve docs in s...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/16137#discussion_r91855221
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -816,6 +833,10 @@ class SparkContext(config: SparkConf) extends Logging {
/**
* Read a text file from HDFS, a local file system (available on all nodes), or any
* Hadoop-supported file system URI, and return it as an RDD of Strings.
+ *
+ * @param path URI to Hadoop-supported file system
--- End diff --
Well, it's a URI representing a path on a supported file system. But the latter fact was already in the docs. The meaningful doc here is that it's the path of the text file!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16137: [SPARK-18708][CORE] Improvement/improve docs in spark co...
Posted by Mironor <gi...@git.apache.org>.
Github user Mironor commented on the issue:
https://github.com/apache/spark/pull/16137
@srowen I see that my rebase didn't go well, sorry for that. Is there a way to revert the last push or it would be better to create another pull request and close this one?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16137: [SPARK-18708][CORE] Improvement/improve docs in s...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/16137#discussion_r91008621
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -119,22 +119,22 @@ class SparkContext(config: SparkConf) extends Logging {
/**
* Alternative constructor that allows setting common Spark properties directly
*
- * @param master Cluster URL to connect to (e.g. mesos://host:port, spark://host:port, local[4]).
- * @param appName A name for your application, to display on the cluster web UI
- * @param conf a [[org.apache.spark.SparkConf]] object specifying other Spark parameters
+ * @param master cluster URL to connect to (e.g. mesos://host:port, spark://host:port, local[4]).
+ * @param appName a name for your application, to display on the cluster web UI
+ * @param conf a `org.apache.spark.SparkConf` object specifying other Spark parameters
*/
def this(master: String, appName: String, conf: SparkConf) =
this(SparkContext.updatedConf(conf, master, appName))
/**
* Alternative constructor that allows setting common Spark properties directly
*
- * @param master Cluster URL to connect to (e.g. mesos://host:port, spark://host:port, local[4]).
- * @param appName A name for your application, to display on the cluster web UI.
- * @param sparkHome Location where Spark is installed on cluster nodes.
- * @param jars Collection of JARs to send to the cluster. These can be paths on the local file
+ * @param master cluster URL to connect to (e.g. mesos://host:port, spark://host:port, local[4]).
+ * @param appName a name for your application, to display on the cluster web UI.
+ * @param sparkHome location where Spark is installed on cluster nodes.
+ * @param jars collection of JARs to send to the cluster. These can be paths on the local file
* system or HDFS, HTTP, HTTPS, or FTP URLs.
- * @param environment Environment variables to set on worker nodes.
+ * @param environment environment variables to set on worker nodes.
--- End diff --
Do we have a rule (or other references) to make them lower-cased? I am worried of similar changes in the future and it might be great if we have a concrete reason here to change so.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16137: [SPARK-18708][CORE] Improvement/improve docs in s...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/16137
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16137: [SPARK-18708][CORE] Improvement/improve docs in s...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/16137#discussion_r92600521
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -2067,8 +2067,8 @@ class SparkContext(config: SparkConf) extends Logging {
}
/**
- * Run a job on all partitions in an RDD and pass the results to a handler function, but take
- * a function of type `Iterator[T] => U` instead of `(TaskContext, Iterator[T]) => U`.
+ * Run a job on all partitions in an RDD and pass the results to a handler function, this time the
--- End diff --
Sure, sounds more natural to me. Instead of "should", say "... each partition additionally takes ..." because it's not optional.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16137: [SPARK-18708][CORE] Improvement/improve docs in s...
Posted by Mironor <gi...@git.apache.org>.
Github user Mironor commented on a diff in the pull request:
https://github.com/apache/spark/pull/16137#discussion_r91949350
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1038,6 +1059,13 @@ class SparkContext(config: SparkConf) extends Logging {
* operation will create many references to the same object.
* If you plan to directly cache, sort, or aggregate Hadoop writable objects, you should first
* copy them using a `map` function.
+ * @param path directory to the input data files, the path can be comma separated paths
+ * as a list of inputs
+ * @param minPartitions suggested minimum number of partitions for the resulting RDD
+ * @param km implicitly found Class of the key associated with the `fm` implicit parameter
--- End diff --
I ran `build/sbt unidoc` as described [here](https://github.com/apache/spark/blob/master/docs/README.md)
I got this output:
```
[info] 100 warnings
[info] Main Java API documentation successful.
```
Searching `/SparkContext.scala` through warnings didn't yield any results. So I guess it's fine.
This is how it looks in Scaladoc: [https://i.imgsafe.org/ea9b9e32bb.png](https://i.imgsafe.org/ea9b9e32bb.png)
Scala's Scaladocs [do contain](http://www.scala-lang.org/api/current/scala/collection/immutable/List.html#sorted[B>:A](implicitord:scala.math.Ordering[B]):Repr) implicit parameters descriptions, otherwise it would be hard to understand method's signature.
I hope this answers your questions, it's up to you to tell me if you want them removed or not.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16137: [SPARK-18708][CORE] Improvement/improve docs in spark co...
Posted by Mironor <gi...@git.apache.org>.
Github user Mironor commented on the issue:
https://github.com/apache/spark/pull/16137
@srowen I reversed obvious comments as well as some minor changes (such as capitalizing). I only left javadoc for some of the non-trivial public api. I can reverse changes for comments where the only diff is the wrapping of references in backquotes.
I'd like to know if you have some idea on whether to use indentation (as in [javadoc](http://www.oracle.com/technetwork/articles/java/index-137868.html)/[scaladoc](http://docs.scala-lang.org/style/scaladoc.html)) as well as what type of character to use when linking a reference (backquote or brackets?), I could modify Spark code style documentation if if's different from javadoc.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16137: [SPARK-18708][CORE] Improvement/improve docs in s...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/16137#discussion_r91010077
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1417,27 +1551,31 @@ class SparkContext(config: SparkConf) extends Logging {
/**
* Add a file to be downloaded with this Spark job on every node.
- * The `path` passed can be either a local file, a file in HDFS (or other Hadoop-supported
- * filesystems), or an HTTP, HTTPS or FTP URI. To access the file in Spark jobs,
- * use `SparkFiles.get(fileName)` to find its download location.
+ *
+ * @param path can be either a local file, a file in HDFS (or other Hadoop-supported
+ * filesystems), or an HTTP, HTTPS or FTP URI. To access the file in Spark jobs,
+ * use `SparkFiles.get(fileName)` to find its download location.
*/
def addFile(path: String): Unit = {
addFile(path, false)
}
/**
- * Returns a list of file paths that are added to resources.
+ * A list of file paths that are added to resources.
--- End diff --
For me, I personally think we just better leave them or duplicate the description into `@return`. I think I am not supposed to decide this. cc @srowen.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16137: [SPARK-18708][CORE] Improvement/improve docs in s...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/16137#discussion_r91855254
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -956,24 +976,24 @@ class SparkContext(config: SparkConf) extends Logging {
}
/**
- * Get an RDD for a Hadoop-readable dataset from a Hadoop JobConf given its InputFormat and other
- * necessary info (e.g. file name for a filesystem-based dataset, table name for HyperTable),
- * using the older MapReduce API (`org.apache.hadoop.mapred`).
+ * Get an RDD for a Hadoop-readable dataset from a Hadoop `JobConf` given its `InputFormat`
+ * and other necessary info (e.g. file name for a filesystem-based dataset, table name
+ * for HyperTable), using the older MapReduce API (`org.apache.hadoop.mapred`).
*
- * @param conf JobConf for setting up the dataset. Note: This will be put into a Broadcast.
+ * @note Because Hadoop's `RecordReader` class re-uses the same Writable object for each
+ * record, directly caching the returned RDD or directly passing it to an aggregation
+ * or shuffle operation will create many references to the same object.
+ * If you plan to directly cache, sort, or aggregate Hadoop writable objects, you
+ * should first copy them using a `map` function.
+ * @param conf `JobConf` for setting up the dataset. Note: This will be put into a Broadcast.
* Therefore if you plan to reuse this conf to create multiple RDDs, you need to make
* sure you won't modify the conf. A safe approach is always creating a new conf for
* a new RDD.
* @param inputFormatClass Class of the InputFormat
* @param keyClass Class of the keys
* @param valueClass Class of the values
- * @param minPartitions Minimum number of Hadoop Splits to generate.
- *
--- End diff --
As before, don't move this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16137: [SPARK-18708][CORE] Improvement/improve docs in s...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/16137#discussion_r91008657
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -119,22 +119,22 @@ class SparkContext(config: SparkConf) extends Logging {
/**
* Alternative constructor that allows setting common Spark properties directly
*
- * @param master Cluster URL to connect to (e.g. mesos://host:port, spark://host:port, local[4]).
- * @param appName A name for your application, to display on the cluster web UI
- * @param conf a [[org.apache.spark.SparkConf]] object specifying other Spark parameters
+ * @param master cluster URL to connect to (e.g. mesos://host:port, spark://host:port, local[4]).
+ * @param appName a name for your application, to display on the cluster web UI
+ * @param conf a `org.apache.spark.SparkConf` object specifying other Spark parameters
*/
def this(master: String, appName: String, conf: SparkConf) =
this(SparkContext.updatedConf(conf, master, appName))
/**
* Alternative constructor that allows setting common Spark properties directly
*
- * @param master Cluster URL to connect to (e.g. mesos://host:port, spark://host:port, local[4]).
- * @param appName A name for your application, to display on the cluster web UI.
- * @param sparkHome Location where Spark is installed on cluster nodes.
- * @param jars Collection of JARs to send to the cluster. These can be paths on the local file
+ * @param master cluster URL to connect to (e.g. mesos://host:port, spark://host:port, local[4]).
+ * @param appName a name for your application, to display on the cluster web UI.
+ * @param sparkHome location where Spark is installed on cluster nodes.
+ * @param jars collection of JARs to send to the cluster. These can be paths on the local file
* system or HDFS, HTTP, HTTPS, or FTP URLs.
- * @param environment Environment variables to set on worker nodes.
+ * @param environment environment variables to set on worker nodes.
--- End diff --
I am fine if there is not too if there can be coherent and this can be decided here too.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16137: [SPARK-18708][CORE] Improvement/improve docs in s...
Posted by Mironor <gi...@git.apache.org>.
Github user Mironor commented on a diff in the pull request:
https://github.com/apache/spark/pull/16137#discussion_r91009155
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1401,8 +1532,11 @@ class SparkContext(config: SparkConf) extends Logging {
/**
* Broadcast a read-only variable to the cluster, returning a
- * [[org.apache.spark.broadcast.Broadcast]] object for reading it in distributed functions.
+ * `org.apache.spark.broadcast.Broadcast` object for reading it in distributed functions.
--- End diff --
Yes, the initial intent was to make it the same everywhere (either backquotes or brackets)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16137: [SPARK-18708][CORE] Improvement/improve docs in s...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/16137#discussion_r91938202
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -2412,6 +2540,9 @@ object SparkContext extends Logging {
/**
* Find the JAR from which a given class was loaded, to make it easy for users to pass
* their JARs to SparkContext.
+ *
+ * @param cls Class that should be inside of the jar
--- End diff --
Nit, I don't even feel strongly about it: "class" isn't capitalized, except where used as the class name `Class`. Most uses in your new docs are just the noun, not class name. So you might just write "class", or else back-tick it where it can be referred to as a class name.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16137: [SPARK-18708][CORE] Improvement/improve docs in s...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/16137#discussion_r90948767
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -224,8 +224,8 @@ class SparkContext(config: SparkConf) extends Logging {
private[spark] def conf: SparkConf = _conf
/**
- * Return a copy of this SparkContext's configuration. The configuration ''cannot'' be
- * changed at runtime.
+ * @return copy of this `SparkContext`'s configuration. The configuration ''cannot'' be
+ * changed at runtime.
--- End diff --
This is fine but you don't need a continuation indent.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16137: [SPARK-18708][CORE] Improvement/improve docs in s...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/16137#discussion_r92132133
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1973,7 +2067,12 @@ class SparkContext(config: SparkConf) extends Logging {
}
/**
- * Run a job on all partitions in an RDD and pass the results to a handler function.
+ * Run a job on all partitions in an RDD and pass the results to a handler function, but take
+ * a function of type `Iterator[T] => U` instead of `(TaskContext, Iterator[T]) => U`.
--- End diff --
I think this one should change too -- instead of talking about the types (which are already in the signature) refer to the salient difference here, that this one doesn't take a function that takes a TaskContext.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16137: [SPARK-18708][CORE] Improvement/improve docs in spark co...
Posted by Mironor <gi...@git.apache.org>.
Github user Mironor commented on the issue:
https://github.com/apache/spark/pull/16137
Hi @srowen , it's quite hard to directly jump on something like "refactoring core scheduling logic", I thought fixing small things would be a good start. To make it simple I limited all the changes to only one file, I also took DAGSceduler.scala as an example for what documenting things should look like (as currently SparkContext.scala is a mix of different document styles, but maybe it's all fine). That's where the continuation indent comes from.
In any case thanks for taking your time to review this PR, I would really appreciate if you could tell me if it's worth removing obvious or insignificant (like de-capitalizing sentences) comments to make this PR less overwhelming and worthwhile of taking a review.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16137: [SPARK-18708][CORE] Improvement/improve docs in s...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/16137#discussion_r91855239
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -816,6 +833,10 @@ class SparkContext(config: SparkConf) extends Logging {
/**
* Read a text file from HDFS, a local file system (available on all nodes), or any
* Hadoop-supported file system URI, and return it as an RDD of Strings.
+ *
+ * @param path URI to Hadoop-supported file system
+ * @param minPartitions suggested value of the minimal splitting number for input data
--- End diff --
I think the doc string is hard to understand. "Suggested minimum number of partitions for the resulting RDD"?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16137: [SPARK-18708][CORE] Improvement/improve docs in s...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/16137#discussion_r91950169
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1038,6 +1059,13 @@ class SparkContext(config: SparkConf) extends Logging {
* operation will create many references to the same object.
* If you plan to directly cache, sort, or aggregate Hadoop writable objects, you should first
* copy them using a `map` function.
+ * @param path directory to the input data files, the path can be comma separated paths
+ * as a list of inputs
+ * @param minPartitions suggested minimum number of partitions for the resulting RDD
+ * @param km implicitly found Class of the key associated with the `fm` implicit parameter
--- End diff --
The thing is that callers won't be specifying these. They're just an implementation detail related to various generic types. If anything, it's the generic types that should be documented, though they're kind of obvious.
These things aren't Classes, but ClassTags. fm is the parameter; it's not something related to fm. Really, fm is related to F.
If this is all we can say about these types I'd remove them, because I'm not sure anyone who doesn't already understand implicit ClassTags is going to be helped by this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org