You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "juliuszsompolski (via GitHub)" <gi...@apache.org> on 2023/06/26 16:34:26 UTC
[GitHub] [spark] juliuszsompolski opened a new pull request, #41742: [SPARK-44195] Add JobTag APIs to SparkR SparkContext
juliuszsompolski opened a new pull request, #41742:
URL: https://github.com/apache/spark/pull/41742
### What changes were proposed in this pull request?
Add APIs from https://github.com/apache/spark/pull/41440 to SparkR:
* addJobTag(tag)
* removeJobTag(tag)
* getJobTags()
* clearJobTags()
* cancelJobsWithTag()
* setInterruptOnCancel(tag)
### Why are the changes needed?
SparkR parity.
### Does this PR introduce _any_ user-facing change?
Yes, introduce the APIs introduced in Scala in https://github.com/apache/spark/pull/41440 to SparkR
### How was this patch tested?
Added test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #41742: [SPARK-44195] Add JobTag APIs to SparkR SparkContext
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #41742:
URL: https://github.com/apache/spark/pull/41742#discussion_r1244038248
##########
core/src/main/scala/org/apache/spark/SparkContext.scala:
##########
@@ -2547,7 +2547,7 @@ class SparkContext(config: SparkConf) extends Logging {
/**
* Cancel active jobs that have the specified tag. See `org.apache.spark.SparkContext.addJobTag`.
*
- * @param tag The tag to be added. Cannot contain ',' (comma) character.
+ * @param tag The tag to be cancelled. Cannot contain ',' (comma) character.
Review Comment:
Mind fixing `JavaSparkContext.cancelJobsWithTag` too since we're here? I just copied and pasted it 😂
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] juliuszsompolski commented on pull request #41742: [SPARK-44195] Add JobTag APIs to SparkR SparkContext
Posted by "juliuszsompolski (via GitHub)" <gi...@apache.org>.
juliuszsompolski commented on PR #41742:
URL: https://github.com/apache/spark/pull/41742#issuecomment-1609544207
```
> addJobTag("foo")
> getJobTags()
Java ref type scala.collection.convert.Wrappers$SetWrapper id 7
```
will need to investigate how to unwrap it to an R type...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] juliuszsompolski commented on a diff in pull request #41742: [SPARK-44195] Add JobTag APIs to SparkR SparkContext
Posted by "juliuszsompolski (via GitHub)" <gi...@apache.org>.
juliuszsompolski commented on code in PR #41742:
URL: https://github.com/apache/spark/pull/41742#discussion_r1243835719
##########
core/src/main/scala/org/apache/spark/api/r/SerDe.scala:
##########
@@ -404,6 +404,22 @@ private[spark] object SerDe {
writeKeyValue(dos, k1.asInstanceOf[Object], v1.asInstanceOf[Object], jvmObjectTracker)
}
+ // Handle Set - write as list
+ case v: java.util.Set[_] =>
+ writeType(dos, "list")
+ writeInt(dos, v.size)
+ val iter = v.iterator
+ while (iter.hasNext) {
+ val elem = iter.next()
+ writeObject(dos, elem.asInstanceOf[Object], jvmObjectTracker)
+ }
+ case v: scala.collection.Set[_] =>
+ writeType(dos, "list")
+ writeInt(dos, v.size)
+ v.foreach { elem =>
+ writeObject(dos, elem.asInstanceOf[Object], jvmObjectTracker)
+ }
Review Comment:
Added SerDe for Set, so that getJobTags work:
```
> getJobTags()
list()
> addJobTag("a")
> getJobTags()
[[1]]
[1] "a"
> clearJobTags()
> getJobTags()
list()
> addJobTag("b")
> addJobTag("c")
> getJobTags()
[[1]]
[1] "b"
[[2]]
[1] "c"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] juliuszsompolski commented on a diff in pull request #41742: [SPARK-44195] Add JobTag APIs to SparkR SparkContext
Posted by "juliuszsompolski (via GitHub)" <gi...@apache.org>.
juliuszsompolski commented on code in PR #41742:
URL: https://github.com/apache/spark/pull/41742#discussion_r1242464201
##########
R/pkg/R/sparkR.R:
##########
@@ -542,6 +542,89 @@ cancelJobGroup <- function(groupId) {
invisible(callJMethod(sc, "cancelJobGroup", groupId))
}
+#' Set the behavior of job cancellation from jobs started in this thread.
+#'
+#' @param interruptOnCancel If true, then job cancellation will result in `Thread.interrupt()`
+#' being called on the job's executor threads. This is useful to help ensure that the tasks
+#' are actually stopped in a timely manner, but is off by default due to HDFS-1208, where HDFS
+#' may respond to Thread.interrupt() by marking nodes as dead.
+#' @rdname setInterruptOnCancel
+#' @name setInterruptOnCancel
+#' @examples
+#'\dontrun{
+#' sparkR.session()
+#' setInterruptOnCancel(true)
+#'}
+#' @note cancelJobGroup since 3.5.0
+setInterruptOnCancel <- function(interruptOnCancel) {
+ sc <- getSparkContext()
+ invisible(callJMethod(sc, "setInterruptOnCancel", interruptOnCancel))
+}
+
+#' Add a tag to be assigned to all the jobs started by this thread.
+#'
+#' @param tag The tag to be added. Cannot contain ',' (comma) character.
+#' @rdname addJobTAg
+#' @name addJobTag
+#' @examples
+#'\dontrun{
+#' sparkR.session()
+#' addJobTag("myJobTag")
+#'}
+#' @note addJobTag since 3.5.0
+addJobTag <- function(tag) {
+ sc <- getSparkContext()
+ invisible(callJMethod(sc, "addJobTag", tag))
+}
+
+#' Remove a tag previously added to be assigned to all the jobs started by this thread.
+#' Noop if such a tag was not added earlier.
+#'
+#' @param tag The tag to be removed. Cannot contain ',' (comma) character.
+#' @rdname removeJobTAg
+#' @name removeJobTag
+#' @examples
+#'\dontrun{
+#' sparkR.session()
+#' removeJobTag("myJobTag")
+#'}
+#' @note cancelJobGroup since 3.5.0
+removeJobTag <- function(tag) {
+ sc <- getSparkContext()
+ invisible(callJMethod(sc, "removeJobTag", tag))
+}
+
+#' Get the tags that are currently set to be assigned to all the jobs started by this thread.
+#'
+#' @rdname getJobTags
+#' @name getJobTags
+#' @examples
+#'\dontrun{
+#' sparkR.session()
+#' tags <- getJobTags()
+#'}
+#' @note getJobTags since 3.5.0
+getJobTags <- function() {
+ sc <- getSparkContext()
+ invisible(callJMethod(sc, "getJobTags"))
+ # TODO: how to return the Scala Set to R??
Review Comment:
@falaki could you help me figure out how getJobTags, which returns a Set[String] in scala, could be made to return an R collection? Or could this API be ignored from R (the main use case is to be able to addJobTag, and then be able to cancelJobsWithTag; the use cases for getting job tags are limited)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] juliuszsompolski commented on a diff in pull request #41742: [SPARK-44195][R] Add JobTag APIs to SparkR SparkContext
Posted by "juliuszsompolski (via GitHub)" <gi...@apache.org>.
juliuszsompolski commented on code in PR #41742:
URL: https://github.com/apache/spark/pull/41742#discussion_r1244102747
##########
core/src/main/scala/org/apache/spark/SparkContext.scala:
##########
@@ -889,6 +893,7 @@ class SparkContext(config: SparkConf) extends Logging {
Option(getLocalProperty(SparkContext.SPARK_JOB_TAGS))
.map(_.split(SparkContext.SPARK_JOB_TAGS_SEP).toSet)
.getOrElse(Set())
+ .filter(!_.isEmpty) // empty string tag should not happen, but be defensive
Review Comment:
... defensively fix the bug twice :-)
##########
core/src/main/scala/org/apache/spark/SparkContext.scala:
##########
@@ -877,7 +877,11 @@ class SparkContext(config: SparkConf) extends Logging {
SparkContext.throwIfInvalidTag(tag)
val existingTags = getJobTags()
val newTags = (existingTags - tag).mkString(SparkContext.SPARK_JOB_TAGS_SEP)
- setLocalProperty(SparkContext.SPARK_JOB_TAGS, newTags)
+ if (newTags.isEmpty) {
+ clearJobTags()
+ } else {
Review Comment:
the new R test found a bug so fixed it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] juliuszsompolski commented on a diff in pull request #41742: [SPARK-44195] Add JobTag APIs to SparkR SparkContext
Posted by "juliuszsompolski (via GitHub)" <gi...@apache.org>.
juliuszsompolski commented on code in PR #41742:
URL: https://github.com/apache/spark/pull/41742#discussion_r1243624154
##########
core/src/main/scala/org/apache/spark/SparkContext.scala:
##########
@@ -2547,7 +2547,7 @@ class SparkContext(config: SparkConf) extends Logging {
/**
* Cancel active jobs that have the specified tag. See `org.apache.spark.SparkContext.addJobTag`.
*
- * @param tag The tag to be added. Cannot contain ',' (comma) character.
+ * @param tag The tag to be cancelled. Cannot contain ',' (comma) character.
Review Comment:
piggy back this typy that I noticed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #41742: [SPARK-44195] Add JobTag APIs to SparkR SparkContext
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #41742:
URL: https://github.com/apache/spark/pull/41742#discussion_r1244047093
##########
core/src/main/scala/org/apache/spark/api/r/SerDe.scala:
##########
@@ -404,6 +404,22 @@ private[spark] object SerDe {
writeKeyValue(dos, k1.asInstanceOf[Object], v1.asInstanceOf[Object], jvmObjectTracker)
}
+ // Handle Set - write as list
+ case v: java.util.Set[_] =>
+ writeType(dos, "list")
+ writeInt(dos, v.size)
+ val iter = v.iterator
+ while (iter.hasNext) {
+ val elem = iter.next()
+ writeObject(dos, elem.asInstanceOf[Object], jvmObjectTracker)
+ }
+ case v: scala.collection.Set[_] =>
+ writeType(dos, "list")
+ writeInt(dos, v.size)
+ v.foreach { elem =>
+ writeObject(dos, elem.asInstanceOf[Object], jvmObjectTracker)
+ }
Review Comment:
@juliuszsompolski I think I would just avoid adding this ser/de here but add a method such as `org.apache.spark.api.r.RUtils.getJobTags(sc: SparkContext): Array[String] = sc.getJobTags().toArray`, and invoke this in `context.R`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #41742: [SPARK-44195][R] Add JobTag APIs to SparkR SparkContext
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #41742: [SPARK-44195][R] Add JobTag APIs to SparkR SparkContext
URL: https://github.com/apache/spark/pull/41742
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] juliuszsompolski commented on a diff in pull request #41742: [SPARK-44195] Add JobTag APIs to SparkR SparkContext
Posted by "juliuszsompolski (via GitHub)" <gi...@apache.org>.
juliuszsompolski commented on code in PR #41742:
URL: https://github.com/apache/spark/pull/41742#discussion_r1243835719
##########
core/src/main/scala/org/apache/spark/api/r/SerDe.scala:
##########
@@ -404,6 +404,22 @@ private[spark] object SerDe {
writeKeyValue(dos, k1.asInstanceOf[Object], v1.asInstanceOf[Object], jvmObjectTracker)
}
+ // Handle Set - write as list
+ case v: java.util.Set[_] =>
+ writeType(dos, "list")
+ writeInt(dos, v.size)
+ val iter = v.iterator
+ while (iter.hasNext) {
+ val elem = iter.next()
+ writeObject(dos, elem.asInstanceOf[Object], jvmObjectTracker)
+ }
+ case v: scala.collection.Set[_] =>
+ writeType(dos, "list")
+ writeInt(dos, v.size)
+ v.foreach { elem =>
+ writeObject(dos, elem.asInstanceOf[Object], jvmObjectTracker)
+ }
Review Comment:
Added SerDe for Set, so that getJobTags work:
```
> getJobTags()
list()
> addJobTag("a")
> getJobTags()
[[1]]
[1] "a"
> clearJobTags()
> getJobTags()
list()
> addJobTag("b")
> addJobTag("c")
> getJobTags()
[[1]]
[1] "b"
[[2]]
[1] "c"
```
Could someone knowing R approve if it's ok to return as list like that, or is there a better R type that this should be serialized and then deserialized in deserialize.R to?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] falaki commented on a diff in pull request #41742: [SPARK-44195] Add JobTag APIs to SparkR SparkContext
Posted by "falaki (via GitHub)" <gi...@apache.org>.
falaki commented on code in PR #41742:
URL: https://github.com/apache/spark/pull/41742#discussion_r1242472329
##########
R/pkg/R/sparkR.R:
##########
@@ -542,6 +542,89 @@ cancelJobGroup <- function(groupId) {
invisible(callJMethod(sc, "cancelJobGroup", groupId))
}
+#' Set the behavior of job cancellation from jobs started in this thread.
+#'
+#' @param interruptOnCancel If true, then job cancellation will result in `Thread.interrupt()`
+#' being called on the job's executor threads. This is useful to help ensure that the tasks
+#' are actually stopped in a timely manner, but is off by default due to HDFS-1208, where HDFS
+#' may respond to Thread.interrupt() by marking nodes as dead.
+#' @rdname setInterruptOnCancel
+#' @name setInterruptOnCancel
+#' @examples
+#'\dontrun{
+#' sparkR.session()
+#' setInterruptOnCancel(true)
+#'}
+#' @note cancelJobGroup since 3.5.0
+setInterruptOnCancel <- function(interruptOnCancel) {
+ sc <- getSparkContext()
+ invisible(callJMethod(sc, "setInterruptOnCancel", interruptOnCancel))
+}
+
+#' Add a tag to be assigned to all the jobs started by this thread.
+#'
+#' @param tag The tag to be added. Cannot contain ',' (comma) character.
+#' @rdname addJobTAg
+#' @name addJobTag
+#' @examples
+#'\dontrun{
+#' sparkR.session()
+#' addJobTag("myJobTag")
+#'}
+#' @note addJobTag since 3.5.0
+addJobTag <- function(tag) {
+ sc <- getSparkContext()
+ invisible(callJMethod(sc, "addJobTag", tag))
+}
+
+#' Remove a tag previously added to be assigned to all the jobs started by this thread.
+#' Noop if such a tag was not added earlier.
+#'
+#' @param tag The tag to be removed. Cannot contain ',' (comma) character.
+#' @rdname removeJobTAg
+#' @name removeJobTag
+#' @examples
+#'\dontrun{
+#' sparkR.session()
+#' removeJobTag("myJobTag")
+#'}
+#' @note cancelJobGroup since 3.5.0
+removeJobTag <- function(tag) {
+ sc <- getSparkContext()
+ invisible(callJMethod(sc, "removeJobTag", tag))
+}
+
+#' Get the tags that are currently set to be assigned to all the jobs started by this thread.
+#'
+#' @rdname getJobTags
+#' @name getJobTags
+#' @examples
+#'\dontrun{
+#' sparkR.session()
+#' tags <- getJobTags()
+#'}
+#' @note getJobTags since 3.5.0
+getJobTags <- function() {
+ sc <- getSparkContext()
+ invisible(callJMethod(sc, "getJobTags"))
+ # TODO: how to return the Scala Set to R??
Review Comment:
It may be hard to return a perfect data structure from this method. Who is calling this? The caller can convert the data type from an Array to a Set.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] juliuszsompolski commented on pull request #41742: [SPARK-44195][R] Add JobTag APIs to SparkR SparkContext
Posted by "juliuszsompolski (via GitHub)" <gi...@apache.org>.
juliuszsompolski commented on PR #41742:
URL: https://github.com/apache/spark/pull/41742#issuecomment-1610964209
CI finished with 1 unrelated flake:
https://github.com/juliuszsompolski/apache-spark/actions/runs/5392816496/jobs/9792933924
```
[info] - SPARK-38187: Run SparkPi Jobs with minMemory *** FAILED *** (3 minutes, 2 seconds)
[info] The code passed to eventually never returned normally. Attempted 190 times over 3.015093976666667 minutes. Last failure message: 0 did not equal 2. (VolcanoTestsSuite.scala:331)
...
[info] *** 1 TEST FAILED ***
[error] Failed tests:
[error] org.apache.spark.deploy.k8s.integrationtest.VolcanoSuite
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #41742: [SPARK-44195][R] Add JobTag APIs to SparkR SparkContext
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #41742:
URL: https://github.com/apache/spark/pull/41742#issuecomment-1617064270
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org