You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/04/28 13:29:32 UTC
[GitHub] [spark] WeichenXu123 opened a new pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
WeichenXu123 opened a new pull request #28395:
URL: https://github.com/apache/spark/pull/28395
### What changes were proposed in this pull request?
I add a new API in pyspark RDD class:
def collectWithJobGroup(self, groupId, description, interruptOnCancel=False)
This API do the same thing with `rdd.collect`, but it can specify the job group when do collect.
The purpose of adding this API is, if we use:
```
sc.setJobGroup("group-id...")
rdd.collect()
```
The `setJobGroup` API in pyspark won't work correctly. This related to a bug discussed in
https://issues.apache.org/jira/browse/SPARK-31549
### Why are the changes needed?
Fix bug.
### Does this PR introduce any user-facing change?
A develop API in pyspark: `pyspark.RDD. collectWithJobGroup`
### How was this patch tested?
Unit test.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-621523265
**[Test build #122095 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122095/testReport)** for PR 28395 at commit [`91cf1c6`](https://github.com/apache/spark/commit/91cf1c6f48e95ae7344ec09399e5005be1f9d78f).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-621751787
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-622199424
Merged to master and branch-3.0. Thanks @mengxr, @WeichenXu123 and @dongjoon-hyun.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #28395:
URL: https://github.com/apache/spark/pull/28395#discussion_r417002009
##########
File path: python/pyspark/tests/test_rdd.py
##########
@@ -814,6 +814,63 @@ def assert_request_contents(exec_reqs, task_reqs):
rddWithoutRp = self.sc.parallelize(range(10))
self.assertEqual(rddWithoutRp.getResourceProfile(), None)
+ def test_collect_with_job_group(self):
Review comment:
Shall we mimic the test case at https://github.com/apache/spark/pull/24898/files#diff-896aaa74767987ed14f1ef9fcb324cefR84?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-621566282
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-621565106
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-621138777
**[Test build #122059 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122059/testReport)** for PR 28395 at commit [`481bba6`](https://github.com/apache/spark/commit/481bba62ce62a13f23ce153e54e8a5f56f6059c2).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-622229256
Thank you. The follow-up looks good. BTW, FYI, `branch-3.0` UT has been broken by another commit.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-622227189
Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-620614897
**[Test build #121999 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121999/testReport)** for PR 28395 at commit [`1ead01d`](https://github.com/apache/spark/commit/1ead01db37e214be60fef8f9c1056661e24ddd15).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-622227485
Ah, it's branch-3.0 only. Let me just hotfix in branch-3.0 only.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #28395:
URL: https://github.com/apache/spark/pull/28395#discussion_r417002980
##########
File path: core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala
##########
@@ -168,6 +168,23 @@ private[spark] object PythonRDD extends Logging {
serveIterator(rdd.collect().iterator, s"serve RDD ${rdd.id}")
}
+ /**
+ * A helper function to collect an RDD as an iterator, then serve it via socket.
+ * This method is similar with `PythonRDD.collectAndServe`, but user can specify job group id,
+ * job description, and interruptOnCancel option.
+ *
+ * Note: This method are temporary, might be removed in future.
Review comment:
We can use Unstable mark too.
##########
File path: core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala
##########
@@ -168,6 +168,23 @@ private[spark] object PythonRDD extends Logging {
serveIterator(rdd.collect().iterator, s"serve RDD ${rdd.id}")
}
+ /**
+ * A helper function to collect an RDD as an iterator, then serve it via socket.
+ * This method is similar with `PythonRDD.collectAndServe`, but user can specify job group id,
+ * job description, and interruptOnCancel option.
+ *
+ * Note: This method are temporary, might be removed in future.
Review comment:
We can use Unstable mark instead.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #28395:
URL: https://github.com/apache/spark/pull/28395#discussion_r417002980
##########
File path: core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala
##########
@@ -168,6 +168,23 @@ private[spark] object PythonRDD extends Logging {
serveIterator(rdd.collect().iterator, s"serve RDD ${rdd.id}")
}
+ /**
+ * A helper function to collect an RDD as an iterator, then serve it via socket.
+ * This method is similar with `PythonRDD.collectAndServe`, but user can specify job group id,
+ * job description, and interruptOnCancel option.
+ *
+ * Note: This method are temporary, might be removed in future.
Review comment:
We can use Unstable mark as well.
##########
File path: core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala
##########
@@ -168,6 +168,23 @@ private[spark] object PythonRDD extends Logging {
serveIterator(rdd.collect().iterator, s"serve RDD ${rdd.id}")
}
+ /**
+ * A helper function to collect an RDD as an iterator, then serve it via socket.
+ * This method is similar with `PythonRDD.collectAndServe`, but user can specify job group id,
+ * job description, and interruptOnCancel option.
+ *
+ * Note: This method are temporary, might be removed in future.
Review comment:
We can use Unstable mark instead as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-621664205
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-620936871
**[Test build #122023 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122023/testReport)** for PR 28395 at commit [`481bba6`](https://github.com/apache/spark/commit/481bba62ce62a13f23ce153e54e8a5f56f6059c2).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-622231583
Ah, it was already commented at https://github.com/apache/spark/pull/28194 :-)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-621221120
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-620979311
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-621565816
**[Test build #122100 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122100/testReport)** for PR 28395 at commit [`91cf1c6`](https://github.com/apache/spark/commit/91cf1c6f48e95ae7344ec09399e5005be1f9d78f).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #28395:
URL: https://github.com/apache/spark/pull/28395#discussion_r417511976
##########
File path: python/pyspark/rdd.py
##########
@@ -877,6 +877,19 @@ def collect(self):
sock_info = self.ctx._jvm.PythonRDD.collectAndServe(self._jrdd.rdd())
return list(_load_from_socket(sock_info, self._jrdd_deserializer))
+ def collectWithJobGroup(self, groupId, description, interruptOnCancel=False):
+ """
+ .. note:: Experimental
+
+ When collect rdd, use this method to specify job group.
+
+ .. versionadded:: 3.0.0
Review comment:
If then, it would be great if we have @HyukjinKwon's [background explanation](https://github.com/apache/spark/pull/28395#pullrequestreview-402300959) on the PR description.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-620937188
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-620614897
**[Test build #121999 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121999/testReport)** for PR 28395 at commit [`1ead01d`](https://github.com/apache/spark/commit/1ead01db37e214be60fef8f9c1056661e24ddd15).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-620936871
**[Test build #122023 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122023/testReport)** for PR 28395 at commit [`481bba6`](https://github.com/apache/spark/commit/481bba62ce62a13f23ce153e54e8a5f56f6059c2).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #28395:
URL: https://github.com/apache/spark/pull/28395#discussion_r417511976
##########
File path: python/pyspark/rdd.py
##########
@@ -877,6 +877,19 @@ def collect(self):
sock_info = self.ctx._jvm.PythonRDD.collectAndServe(self._jrdd.rdd())
return list(_load_from_socket(sock_info, self._jrdd_deserializer))
+ def collectWithJobGroup(self, groupId, description, interruptOnCancel=False):
+ """
+ .. note:: Experimental
+
+ When collect rdd, use this method to specify job group.
+
+ .. versionadded:: 3.0.0
Review comment:
It would be great if we have @HyukjinKwon [explanation](https://github.com/apache/spark/pull/28395#pullrequestreview-402300959) on the PR description.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-621600899
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #28395:
URL: https://github.com/apache/spark/pull/28395#discussion_r417672569
##########
File path: python/pyspark/rdd.py
##########
@@ -877,6 +877,19 @@ def collect(self):
sock_info = self.ctx._jvm.PythonRDD.collectAndServe(self._jrdd.rdd())
return list(_load_from_socket(sock_info, self._jrdd_deserializer))
+ def collectWithJobGroup(self, groupId, description, interruptOnCancel=False):
+ """
+ .. note:: Experimental
+
+ When collect rdd, use this method to specify job group.
+
+ .. versionadded:: 3.0.0
Review comment:
I don't think we will ignore the existing definitions of API stability tags. Might be best to stick to what's documented there in tags ...
Of course we should take it considering the ammended semver but I think it's a rather rubric to consider.
I kind of strongly think we should deprecate/remove it to stick to the standard approach for this one specifically later once we enable the pin-thread mode by default. I also agree with having this APIs as a step to migrate meanwhile.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-621523674
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-621564776
**[Test build #122095 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122095/testReport)** for PR 28395 at commit [`91cf1c6`](https://github.com/apache/spark/commit/91cf1c6f48e95ae7344ec09399e5005be1f9d78f).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-621221135
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/122059/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-620972164
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/122023/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-621221120
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-620972161
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-621600902
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/122100/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-621750505
**[Test build #122120 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122120/testReport)** for PR 28395 at commit [`bdd77fe`](https://github.com/apache/spark/commit/bdd77fecc886a7b94a66c8c3dfd27f8923c22015).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-621663547
**[Test build #122120 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122120/testReport)** for PR 28395 at commit [`bdd77fe`](https://github.com/apache/spark/commit/bdd77fecc886a7b94a66c8c3dfd27f8923c22015).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-622225266
Hi, @HyukjinKwon .
Could you fix Python linter error?
```
./python/pyspark/tests/test_rdd.py:787:5: E303 too many blank lines (2)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-621565106
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-621751787
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #28395:
URL: https://github.com/apache/spark/pull/28395#discussion_r417672569
##########
File path: python/pyspark/rdd.py
##########
@@ -877,6 +877,19 @@ def collect(self):
sock_info = self.ctx._jvm.PythonRDD.collectAndServe(self._jrdd.rdd())
return list(_load_from_socket(sock_info, self._jrdd_deserializer))
+ def collectWithJobGroup(self, groupId, description, interruptOnCancel=False):
+ """
+ .. note:: Experimental
+
+ When collect rdd, use this method to specify job group.
+
+ .. versionadded:: 3.0.0
Review comment:
I dont think we will ignore the existing definitions of API stability tags. Might be best to stick to what's documented there in tags ...
Of course we should take it considering the ammended semver but I think it's a rather rubric to consider when we break.
I kind of strongly think we should deprecate/remove it to stick to the standard approach for this one specifically later when we're able. I also agree with having this APIs as a step to migrate meanwhile.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-622225266
Hi, @HyukjinKwon .
Could you fix Python linter error on `branch-3.0`?
```
./python/pyspark/tests/test_rdd.py:787:5: E303 too many blank lines (2)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #28395:
URL: https://github.com/apache/spark/pull/28395#discussion_r417001144
##########
File path: python/pyspark/rdd.py
##########
@@ -877,6 +877,17 @@ def collect(self):
sock_info = self.ctx._jvm.PythonRDD.collectAndServe(self._jrdd.rdd())
return list(_load_from_socket(sock_info, self._jrdd_deserializer))
+ def collectWithJobGroup(self, groupId, description, interruptOnCancel=False):
+ """
+ When collect rdd, use this method to specify job group.
+
+ .. note:: This method are temporary, might be removed in future.
Review comment:
Let's also add the `versionadded` or `@since`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-620610170
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-620972161
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-621138777
**[Test build #122059 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122059/testReport)** for PR 28395 at commit [`481bba6`](https://github.com/apache/spark/commit/481bba62ce62a13f23ce153e54e8a5f56f6059c2).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-620971989
**[Test build #122023 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122023/testReport)** for PR 28395 at commit [`481bba6`](https://github.com/apache/spark/commit/481bba62ce62a13f23ce153e54e8a5f56f6059c2).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-621663547
**[Test build #122120 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122120/testReport)** for PR 28395 at commit [`bdd77fe`](https://github.com/apache/spark/commit/bdd77fecc886a7b94a66c8c3dfd27f8923c22015).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-621220102
**[Test build #122059 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122059/testReport)** for PR 28395 at commit [`481bba6`](https://github.com/apache/spark/commit/481bba62ce62a13f23ce153e54e8a5f56f6059c2).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-622229256
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-620736970
**[Test build #121999 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121999/testReport)** for PR 28395 at commit [`1ead01d`](https://github.com/apache/spark/commit/1ead01db37e214be60fef8f9c1056661e24ddd15).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-621565816
**[Test build #122100 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122100/testReport)** for PR 28395 at commit [`91cf1c6`](https://github.com/apache/spark/commit/91cf1c6f48e95ae7344ec09399e5005be1f9d78f).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mengxr commented on a change in pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
mengxr commented on a change in pull request #28395:
URL: https://github.com/apache/spark/pull/28395#discussion_r416885587
##########
File path: core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala
##########
@@ -168,6 +168,23 @@ private[spark] object PythonRDD extends Logging {
serveIterator(rdd.collect().iterator, s"serve RDD ${rdd.id}")
}
+ /**
+ * A helper function to collect an RDD as an iterator, then serve it via socket.
+ * This method is similar with `PythonRDD.collectAndServe`, but user can specify job group id,
+ * job description, and interruptOnCancel option.
+ *
+ * Note: This method are temporary, might be removed in future.
Review comment:
Could you use Experimental note?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-620738507
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-622226989
Hm, weird. It was a clean backport. Let me make a fix in the master through branch-3.0 to reduce the diff. Seems it's legitimate anyway.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #28395:
URL: https://github.com/apache/spark/pull/28395#discussion_r417512584
##########
File path: python/pyspark/rdd.py
##########
@@ -877,6 +877,19 @@ def collect(self):
sock_info = self.ctx._jvm.PythonRDD.collectAndServe(self._jrdd.rdd())
return list(_load_from_socket(sock_info, self._jrdd_deserializer))
+ def collectWithJobGroup(self, groupId, description, interruptOnCancel=False):
+ """
+ .. note:: Experimental
+
+ When collect rdd, use this method to specify job group.
+
+ .. versionadded:: 3.0.0
Review comment:
BTW, we cannot remove this API forever. cc @marmbrus
> I will target to deprecate this API in Spark 3.1.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-620610888
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-621139361
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-621230128
@WeichenXu123 the test failure seems legitimate.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #28395:
URL: https://github.com/apache/spark/pull/28395#discussion_r417512584
##########
File path: python/pyspark/rdd.py
##########
@@ -877,6 +877,19 @@ def collect(self):
sock_info = self.ctx._jvm.PythonRDD.collectAndServe(self._jrdd.rdd())
return list(_load_from_socket(sock_info, self._jrdd_deserializer))
+ def collectWithJobGroup(self, groupId, description, interruptOnCancel=False):
+ """
+ .. note:: Experimental
+
+ When collect rdd, use this method to specify job group.
+
+ .. versionadded:: 3.0.0
Review comment:
BTW, we can remove this API since this is Experimental? cc @marmbrus
> I will target to deprecate this API in Spark 3.1.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-620610888
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-620980838
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #28395:
URL: https://github.com/apache/spark/pull/28395#discussion_r417672569
##########
File path: python/pyspark/rdd.py
##########
@@ -877,6 +877,19 @@ def collect(self):
sock_info = self.ctx._jvm.PythonRDD.collectAndServe(self._jrdd.rdd())
return list(_load_from_socket(sock_info, self._jrdd_deserializer))
+ def collectWithJobGroup(self, groupId, description, interruptOnCancel=False):
+ """
+ .. note:: Experimental
+
+ When collect rdd, use this method to specify job group.
+
+ .. versionadded:: 3.0.0
Review comment:
I dont think we will ignore the existing definitions of API stability tags. Might be best to stick to what's documented there ...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-622229256
Thank you. The follow-up looks good. BTW, `branch-3.0` UT is broken now by another commit. :(
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-620738507
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-620610178
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121997/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-621137033
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-620937188
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-621523674
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-621523265
**[Test build #122095 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122095/testReport)** for PR 28395 at commit [`91cf1c6`](https://github.com/apache/spark/commit/91cf1c6f48e95ae7344ec09399e5005be1f9d78f).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #28395:
URL: https://github.com/apache/spark/pull/28395#discussion_r417511361
##########
File path: python/pyspark/rdd.py
##########
@@ -877,6 +877,19 @@ def collect(self):
sock_info = self.ctx._jvm.PythonRDD.collectAndServe(self._jrdd.rdd())
return list(_load_from_socket(sock_info, self._jrdd_deserializer))
+ def collectWithJobGroup(self, groupId, description, interruptOnCancel=False):
+ """
+ .. note:: Experimental
+
+ When collect rdd, use this method to specify job group.
+
+ .. versionadded:: 3.0.0
Review comment:
Do we target this at 3.0.0?
cc @gatorsmile
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-620980838
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-620610170
Build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-621600566
**[Test build #122100 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122100/testReport)** for PR 28395 at commit [`91cf1c6`](https://github.com/apache/spark/commit/91cf1c6f48e95ae7344ec09399e5005be1f9d78f).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-621664205
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-621139361
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-622225266
Hi, @HyukjinKwon .
Could you fix Python linter error on `branch-3.0`?
- https://github.com/apache/spark/commits/branch-3.0
```
./python/pyspark/tests/test_rdd.py:787:5: E303 too many blank lines (2)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-622231048
Thanks for letting me know. I will take a look too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-620980554
**[Test build #122034 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122034/testReport)** for PR 28395 at commit [`481bba6`](https://github.com/apache/spark/commit/481bba62ce62a13f23ce153e54e8a5f56f6059c2).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-621565163
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-621600899
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28395: [SPARK-31549][PYSPARK] Add a develop API invoking collect on Python RDD with user-specified job group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28395:
URL: https://github.com/apache/spark/pull/28395#issuecomment-621566282
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org