[GitHub] [spark] HyukjinKwon commented on a change in pull request #24705: [SPARK-22340][PYTHON] Save localProperties in thread.local

[GitBox <gi...@apache.org>]

HyukjinKwon commented on a change in pull request #24705: [SPARK-22340][PYTHON] Save localProperties in thread.local
URL: https://github.com/apache/spark/pull/24705#discussion_r287626295
 
 

 ##########
 File path: python/pyspark/context.py
 ##########
 @@ -1002,27 +1005,41 @@ def setJobGroup(self, groupId, description, interruptOnCancel=False):
         ensure that the tasks are actually stopped in a timely manner, but is off by default due
         to HDFS-1208, where HDFS may respond to Thread.interrupt() by marking nodes as dead.
         """
-        self._jsc.setJobGroup(groupId, description, interruptOnCancel)
+        self.setLocalProperty("spark.jobGroup.id", groupId)
+        self.setLocalProperty("spark.job.description", description)
+        self.setLocalProperty("spark.job.interruptOnCancel", str(interruptOnCancel))
+
+    def getLocalProperties(self):
+        if not hasattr(self.local, 'local_properties'):
+            return {}
+        return self.local.local_properties
 
     def setLocalProperty(self, key, value):
         """
         Set a local property that affects jobs submitted from this thread, such as the
         Spark fair scheduler pool.
         """
-        self._jsc.setLocalProperty(key, value)
 
 Review comment:
   I think this will remove the support of TaskContext's localProperties access.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org