You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by holdenk <gi...@git.apache.org> on 2016/02/22 22:39:23 UTC

[GitHub] spark pull request: [SPARK-13398][STREAMING] move away from thread...

GitHub user holdenk opened a pull request:

    https://github.com/apache/spark/pull/11310

    [SPARK-13398][STREAMING] move away from thread pool task support

    ## What changes were proposed in this pull request?
    
    Remove old deprecated ThreadPoolExecutor and replace with ExecutionContext using a ForkJoinPool. The downside of this is that scala's ForkJoinPool doesn't give us a way to specify the thread pool name. One other implicit change that happens is the old ExecutionContext would have reported a different default parallelism since it used system parallelism rather than threadpool parallelism (this was likely not intended but also likely not a huge difference).
    
    The previous version of this PR attempted to use an execution context constructed on the ThreadPool (but not the deprecated ThreadPoolExecutor class) so as to keep the ability to have human readable named threads but this reported system parallelism.
    
    ## How was the this patch tested?
    
    unit tests: streaming/testOnly org.apache.spark.streaming.util.*

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/holdenk/spark SPARK-13398-move-away-from-ThreadPoolTaskSupport

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/11310.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #11310
    
----
commit 68df46a1093132ad3d5eef3e7a3d669105083236
Author: Holden Karau <ho...@us.ibm.com>
Date:   2016-02-21T07:08:45Z

    Change FileWriteAheadLogger to not depend on deprecated ThreadPoolContext

commit 9991ebce07b6fabe122719ba1c4690e733297b18
Author: Holden Karau <ho...@us.ibm.com>
Date:   2016-02-21T07:13:40Z

    Remove now unused import

commit f2e6a25ae2581e9b6cc43950c1302c26a8602c2a
Author: Holden Karau <ho...@us.ibm.com>
Date:   2016-02-21T07:19:27Z

    Update the comment to point out if the thread pool is smaller than 8, that is the number of objects kept in memory.

commit ed9ff855871bfef69b39b1b9b791d1ce487093bd
Author: Holden Karau <ho...@us.ibm.com>
Date:   2016-02-22T01:07:56Z

    Use a ForkJoinPool as the base

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-13398][STREAMING] move away from thread...

Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the pull request:

    https://github.com/apache/spark/pull/11310#issuecomment-189573732
  
    If we are going to go through the effort to do a custom factory for the thread names, looking at the future (e.g. scala 2.12) might be worth it to use the Java interface rather than the scala wrappers provided that we don't run into the same reporting incorrect parallelism problem. I'm going to close this for now and investigate that option mid next week once I'm back on my feet.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-13398][STREAMING] move away from thread...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/11310#issuecomment-187400943
  
    **[Test build #51685 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51685/consoleFull)** for PR 11310 at commit [`ed9ff85`](https://github.com/apache/spark/commit/ed9ff855871bfef69b39b1b9b791d1ce487093bd).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-13398][STREAMING] move away from thread...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11310#discussion_r54335512
  
    --- Diff: core/src/main/scala/org/apache/spark/util/ThreadUtils.scala ---
    @@ -156,4 +156,11 @@ private[spark] object ThreadUtils {
             result
         }
       }
    +
    +  /**
    +   * Construct a new ForkJoinPool with a specified max parallelism.
    +   */
    +  def newForkJoinPool(maxThreadNumber: Int): scala.concurrent.forkjoin.ForkJoinPool = {
    --- End diff --
    
    Yes, if it's more than calling a constructor it may be worth it but not to wrap up "new ForkJoinPool" as "newForkJoinPool = new ForkJoinPool"


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-13398][STREAMING] move away from thread...

Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11310#discussion_r54321079
  
    --- Diff: core/src/main/scala/org/apache/spark/util/ThreadUtils.scala ---
    @@ -156,4 +156,11 @@ private[spark] object ThreadUtils {
             result
         }
       }
    +
    +  /**
    +   * Construct a new ForkJoinPool with a specified max parallelism.
    +   */
    +  def newForkJoinPool(maxThreadNumber: Int): scala.concurrent.forkjoin.ForkJoinPool = {
    +    new scala.concurrent.forkjoin.ForkJoinPool(maxThreadNumber)
    --- End diff --
    
    Please add a thread factory to set the thread name


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-13398][STREAMING] move away from thread...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:

    https://github.com/apache/spark/pull/11310#issuecomment-187661137
  
    That seems OK. It's weird that the first approach didn't work and that fork-join works as intended, but hey.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-13398][STREAMING] move away from thread...

Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11310#discussion_r54327633
  
    --- Diff: core/src/main/scala/org/apache/spark/util/ThreadUtils.scala ---
    @@ -156,4 +156,11 @@ private[spark] object ThreadUtils {
             result
         }
       }
    +
    +  /**
    +   * Construct a new ForkJoinPool with a specified max parallelism.
    +   */
    +  def newForkJoinPool(maxThreadNumber: Int): scala.concurrent.forkjoin.ForkJoinPool = {
    +    new scala.concurrent.forkjoin.ForkJoinPool(maxThreadNumber)
    --- End diff --
    
    So, with the scala ForkJoinPool even when provide a thread factory, it sets the thread names based on the "workerNamePrefix" (which is final) inside of "registerWorker" which is called when constructing a new ForkJoinWorkerThread - although with a custom factory we can change the name back after the thread has been constructed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-13398][STREAMING] move away from thread...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/11310#issuecomment-187446826
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51685/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-13398][STREAMING] move away from thread...

Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11310#discussion_r54327631
  
    --- Diff: core/src/main/scala/org/apache/spark/util/ThreadUtils.scala ---
    @@ -156,4 +156,11 @@ private[spark] object ThreadUtils {
             result
         }
       }
    +
    +  /**
    +   * Construct a new ForkJoinPool with a specified max parallelism.
    +   */
    +  def newForkJoinPool(maxThreadNumber: Int): scala.concurrent.forkjoin.ForkJoinPool = {
    --- End diff --
    
    I figured it would be best to put it here with the rest of the ThreadUtils stuff. If we add the custom factory for the name as @zsxwing asked gets complicated enough probably worth keeping here in a utility function.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-13398][STREAMING] move away from thread...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:

    https://github.com/apache/spark/pull/11310#issuecomment-189680177
  
    This seemed like a good change to get rid of the deprecation. The thread name is kind of secondary to me.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-13398][STREAMING] move away from thread...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/11310#issuecomment-187446823
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-13398][STREAMING] move away from thread...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/11310#issuecomment-187446529
  
    **[Test build #51685 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51685/consoleFull)** for PR 11310 at commit [`ed9ff85`](https://github.com/apache/spark/commit/ed9ff855871bfef69b39b1b9b791d1ce487093bd).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-13398][STREAMING] move away from thread...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11310#discussion_r53767796
  
    --- Diff: core/src/main/scala/org/apache/spark/util/ThreadUtils.scala ---
    @@ -156,4 +156,11 @@ private[spark] object ThreadUtils {
             result
         }
       }
    +
    +  /**
    +   * Construct a new ForkJoinPool with a specified max parallelism.
    +   */
    +  def newForkJoinPool(maxThreadNumber: Int): scala.concurrent.forkjoin.ForkJoinPool = {
    --- End diff --
    
    Do we need a method for this, really, vs just calling that constructor?
    can it be imported as a class name?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-13398][STREAMING] move away from thread...

Posted by holdenk <gi...@git.apache.org>.
Github user holdenk closed the pull request at:

    https://github.com/apache/spark/pull/11310


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org