[GitHub] [spark] HyukjinKwon commented on a change in pull request #24946: [SPARK-27234][SS][PYTHON] Use InheritableThreadLocal for current epoch in EpochTracker (to support Python UDFs)

[GitBox <gi...@apache.org>]

HyukjinKwon commented on a change in pull request #24946: [SPARK-27234][SS][PYTHON] Use InheritableThreadLocal for current epoch in EpochTracker (to support Python UDFs)
URL: https://github.com/apache/spark/pull/24946#discussion_r297435022
 
 

 ##########
 File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/EpochTracker.scala
 ##########
 @@ -19,15 +19,24 @@ package org.apache.spark.sql.execution.streaming.continuous
 
 import java.util.concurrent.atomic.AtomicLong
 
+import org.apache.commons.lang3.SerializationUtils
+
 /**
  * Tracks the current continuous processing epoch within a task. Call
  * EpochTracker.getCurrentEpoch to get the current epoch.
  */
 object EpochTracker {
   // The current epoch. Note that this is a shared reference; ContinuousWriteRDD.compute() will
   // update the underlying AtomicLong as it finishes epochs. Other code should only read the value.
-  private val currentEpoch: ThreadLocal[AtomicLong] = new ThreadLocal[AtomicLong] {
-    override def initialValue() = new AtomicLong(-1)
+  private val currentEpoch: InheritableThreadLocal[AtomicLong] = {
+    new InheritableThreadLocal[AtomicLong] {
+      override protected def childValue(parent: AtomicLong): AtomicLong = {
+        // Note: make a clone such that changes in the parent epoch aren't reflected in
+        // those in the children threads. This is required at `ContinuousCoalesceRDD`.
+        SerializationUtils.clone(parent)
 
 Review comment:
   Ah, sure. Thanks!

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org