You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Steve Loughran (JIRA)" <ji...@apache.org> on 2017/08/16 14:50:00 UTC
[jira] [Commented] (SPARK-20952) ParquetFileFormat should forward
TaskContext to its forkjoinpool
[ https://issues.apache.org/jira/browse/SPARK-20952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16128891#comment-16128891 ]
Steve Loughran commented on SPARK-20952:
----------------------------------------
out of curiosity, what "filesystem games" are you playing...they kind of contain assumptions about being cross thread themselves, and have their own pools of workers
> ParquetFileFormat should forward TaskContext to its forkjoinpool
> ----------------------------------------------------------------
>
> Key: SPARK-20952
> URL: https://issues.apache.org/jira/browse/SPARK-20952
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.1.1
> Reporter: Robert Kruszewski
> Priority: Minor
>
> TaskContext is a ThreadLocal as a result when you fork a thread inside your executor task you lose the handle on the original context set by the executor. We should change it to InheritableThreadLocal so we can access it inside thread pools on executors.
> See ParquetFileFormat#readFootersInParallel for example of code that uses thread pools inside the tasks.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org