You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Kapil Singh (Jira)" <ji...@apache.org> on 2023/07/17 11:28:00 UTC
[jira] [Updated] (SPARK-44458) ThreadLocal not being copied to child thread when child thread is reused from pool
[ https://issues.apache.org/jira/browse/SPARK-44458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kapil Singh updated SPARK-44458:
--------------------------------
Description:
Currently setting the Spark configuration using the statement `spark.conf.set("spark.sql.caseSensitive", "true")` and subsequently executing a Spark action in a separate thread. An intermittent observation reveals that occasionally the configuration set in the main thread fails to propagate to the child thread.
!image-2023-07-17-16-56-59-388.png!
!image-2023-07-17-16-57-09-669.png!
This is limitation due to ThreadLocal not being copied to child thread when child thread is reused from pool and only done instead in thread creation or when idle thread has expired KeepAlive in Executor
Refer:
[https://users.scala-lang.org/t/future-executioncontext-and-threadlocal/7675/2]
[https://www.stevenskelton.ca/threadlocal-variables-scala-futures/]
was:
Currently setting the Spark configuration using the statement `spark.conf.set("spark.sql.caseSensitive", "true")` and subsequently executing a Spark action in a separate thread. An intermittent observation reveals that occasionally the configuration set in the main thread fails to propagate to the child thread.
!image-2023-07-17-16-54-26-372.png!
!image-2023-07-17-16-54-36-860.png!
This is limitation due to ThreadLocal not being copied to child thread when child thread is reused from pool and only done instead in thread creation or when idle thread has expired KeepAlive in Executor
Refer:
[https://users.scala-lang.org/t/future-executioncontext-and-threadlocal/7675/2]
[https://www.stevenskelton.ca/threadlocal-variables-scala-futures/]
> ThreadLocal not being copied to child thread when child thread is reused from pool
> -----------------------------------------------------------------------------------
>
> Key: SPARK-44458
> URL: https://issues.apache.org/jira/browse/SPARK-44458
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 3.4.1
> Reporter: Kapil Singh
> Priority: Major
> Attachments: image-2023-07-17-16-56-59-388.png, image-2023-07-17-16-57-09-669.png
>
>
> Currently setting the Spark configuration using the statement `spark.conf.set("spark.sql.caseSensitive", "true")` and subsequently executing a Spark action in a separate thread. An intermittent observation reveals that occasionally the configuration set in the main thread fails to propagate to the child thread.
> !image-2023-07-17-16-56-59-388.png!
>
> !image-2023-07-17-16-57-09-669.png!
> This is limitation due to ThreadLocal not being copied to child thread when child thread is reused from pool and only done instead in thread creation or when idle thread has expired KeepAlive in Executor
> Refer:
> [https://users.scala-lang.org/t/future-executioncontext-and-threadlocal/7675/2]
> [https://www.stevenskelton.ca/threadlocal-variables-scala-futures/]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org