You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@helix.apache.org by lei-xia <gi...@git.apache.org> on 2015/11/08 07:47:46 UTC

[GitHub] helix pull request: [HELIX-613] Fix thread leaking problems in Tas...

GitHub user lei-xia opened a pull request:

    https://github.com/apache/helix/pull/38

    [HELIX-613] Fix thread leaking problems in TaskStateModel.

    Current TaskStateModel creates a thread-pool containing 40 threads for each instance of TaskStateModel, thus it creates 40 threads for each task (partition). Since Job are dynamic resources, the thread pool is not properly shutdown when task has completed (or timeouted, failed, etc). We saw ~10000 threads were created in our production machines.
    
    Also, the timeout timer in each TaskStateModel is not properly cancelled even though the task has completed or failed. The timer consume a thread even though it is not used anymore.
    
    The proposed solution is to use a shared thread pool for all TaskStateModels created from the same TaskStateModelFactory for all regular tasks and timeout tasks.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/lei-xia/helix helix-0.6.x

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/helix/pull/38.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #38
    
----
commit 456ddb0c4a900ee7cdf081777eff9445378df513
Author: Lei Xia <lx...@linkedin.com>
Date:   2015-11-08T06:43:04Z

    [HELIX-613] Fix thread leaking problems in TaskStateModel by sharing one thread pool among all tasks and timeout tasks from TaskStateModels created from the same TaskStateModelFactory.

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] helix pull request: [HELIX-613] Fix thread leaking problems in Tas...

Posted by kishoreg <gi...@git.apache.org>.

Github user kishoreg commented on the pull request:

    https://github.com/apache/helix/pull/38#issuecomment-154778932
  
    LGTM. Any test case to ensure that there is no leak?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] helix pull request: [HELIX-613] Fix thread leaking problems in Tas...

Posted by kanakb <gi...@git.apache.org>.

Github user kanakb commented on the pull request:

    https://github.com/apache/helix/pull/38#issuecomment-154940124
  
    Merged.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] helix pull request: [HELIX-613] Fix thread leaking problems in Tas...

Posted by asfgit <gi...@git.apache.org>.

Github user asfgit closed the pull request at:

    https://github.com/apache/helix/pull/38


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] helix pull request: [HELIX-613] Fix thread leaking problems in Tas...

Posted by lei-xia <gi...@git.apache.org>.

Github user lei-xia commented on the pull request:

    https://github.com/apache/helix/pull/38#issuecomment-154924074
  
    It is little bit tricky to add unit test to verify there is no leaking, also it takes some time to generate and detect leaking, which makes the unit tests takes even longer time (it already takes quite long time to run all unit tests now). 
    
    Having said that, although we do not have unit test on this,  we have already verified this fix on our testing environments, with Helix runs for two days, starting and finishing around ~10000 jobs. We did not see significant amount of threads created as we saw before, and also the thread numbers keeps pretty stable over the time.  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---