You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "juliuszsompolski (via GitHub)" <gi...@apache.org> on 2023/09/29 19:20:52 UTC
[GitHub] [spark] juliuszsompolski opened a new pull request, #43182: [SPARK-44762][CONNECT][CORE] Doc for SparkConnect.addJobTag and Connect SparkSession.addTag
juliuszsompolski opened a new pull request, #43182:
URL: https://github.com/apache/spark/pull/43182
### What changes were proposed in this pull request?
Add more documentation about using tags, similar to how SparkConnect.setJobGroup is documented.
### Why are the changes needed?
Better doc.
### Does this PR introduce _any_ user-facing change?
Yes, better doc.
### How was this patch tested?
Doc only.
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #43182: [SPARK-44762][CONNECT][CORE] Doc for SparkConnect.addJobTag and Connect SparkSession.addTag
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #43182:
URL: https://github.com/apache/spark/pull/43182#discussion_r1341873785
##########
python/pyspark/sql/session.py:
##########
@@ -2010,11 +2010,19 @@ def addTag(self, tag: str) -> None:
"""
Add a tag to be assigned to all the operations started by this thread in this session.
+ Often, a unit of execution in an application consists of multiple Spark executions.
+ Application programmers can use this method to group all those jobs together and give a
+ group tag. The application can use :meth:`SparkSession.interruptTag` to cancel all running
+ executions with this tag.
+
+ There may be multiple tags present at the same time, so different parts of application may
+ use different tags to perform cancellation at different levels of granularity.
+
Review Comment:
They are actually documented as so in rst file, https://github.com/apache/spark/blob/master/python/docs/source/reference/pyspark.sql/spark_session.rst#spark-connect-only
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] juliuszsompolski commented on pull request #43182: [SPARK-44762][CONNECT][CORE] Doc for SparkConnect.addJobTag and Connect SparkSession.addTag
Posted by "juliuszsompolski (via GitHub)" <gi...@apache.org>.
juliuszsompolski commented on PR #43182:
URL: https://github.com/apache/spark/pull/43182#issuecomment-1741376812
cc @hvanhovell @HyukjinKwon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] juliuszsompolski commented on a diff in pull request #43182: [SPARK-44762][CONNECT][CORE] Doc for SparkConnect.addJobTag and Connect SparkSession.addTag
Posted by "juliuszsompolski (via GitHub)" <gi...@apache.org>.
juliuszsompolski commented on code in PR #43182:
URL: https://github.com/apache/spark/pull/43182#discussion_r1341735106
##########
python/pyspark/sql/session.py:
##########
@@ -2010,11 +2010,19 @@ def addTag(self, tag: str) -> None:
"""
Add a tag to be assigned to all the operations started by this thread in this session.
+ Often, a unit of execution in an application consists of multiple Spark executions.
+ Application programmers can use this method to group all those jobs together and give a
+ group tag. The application can use :meth:`SparkSession.interruptTag` to cancel all running
+ executions with this tag.
+
+ There may be multiple tags present at the same time, so different parts of application may
+ use different tags to perform cancellation at different levels of granularity.
+
Review Comment:
qq @HyukjinKwon : since this function is available only for Connect, but documented for regular session, shouldn't the doc say so? (same for other connect only functions)
Also, I wanted to add an example, but here examples will not work, because they would result in this exception (and pyspark testing actually checks if examples are runnable)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #43182: [SPARK-44762][CONNECT][CORE] Doc for SparkConnect.addJobTag and Connect SparkSession.addTag
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #43182:
URL: https://github.com/apache/spark/pull/43182#issuecomment-1741749340
I think the test failures look unrelated.
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #43182: [SPARK-44762][CONNECT][CORE] Doc for SparkConnect.addJobTag and Connect SparkSession.addTag
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #43182: [SPARK-44762][CONNECT][CORE] Doc for SparkConnect.addJobTag and Connect SparkSession.addTag
URL: https://github.com/apache/spark/pull/43182
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org