You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by manishgupta88 <gi...@git.apache.org> on 2018/03/21 09:31:51 UTC
[GitHub] carbondata pull request #2086: [CARBONDATA-2181] Thread Leak during compacti...
GitHub user manishgupta88 opened a pull request:
https://github.com/apache/carbondata/pull/2086
[CARBONDATA-2181] Thread Leak during compaction processing on restructured table
**Problem**
Thread leak in compaction operation
**Analysis**
Compaction uses both query and data loading processes. During data laod operation during compaction new threads are spawned in sorting, merger and
data writer step using executor service. These threads are not getting closed in case comapciton fails or the operation is killed from spark UI
as observed by taking a thread dump after compaction failure.
**Fix**
Add a task completion listener in each compaction task which will close all the executor service instances as well as clean any other system
resources to prevent thread leak
- [ ] Any interfaces changed?
No
- [ ] Any backward compatibility impacted?
No
- [ ] Document update required?
No
- [ ] Testing done
Manually Verified in cluster testing
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
NA
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/manishgupta88/carbondata compaction_thread_leak_fix
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/2086.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2086
----
commit ca8e8b1ec81c1b2dda10ff2a24bede17aebb112e
Author: manishgupta88 <to...@...>
Date: 2018-03-21T06:38:01Z
Problem
Thread leak in compaction operation
Analysis
Compaction uses both query and data loading processes. During data laod operation during compaction new threads are spawned in sorting, merger and
data writer step using executor service. These threads are not getting closed in case comapciton fails or the operation is killed from spark UI
as observed by taking a thread dump after compaction failure.
Fix
Add a task completion listener in each compaction task which will close all the executor service instances as well as clean any other system
resources to prevent thread leak
----
---
[GitHub] carbondata issue #2086: [CARBONDATA-2181] Thread Leak during compaction proc...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2086
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3268/
---
[GitHub] carbondata issue #2086: [CARBONDATA-2181] Thread Leak during compaction proc...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2086
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3245/
---
[GitHub] carbondata issue #2086: [CARBONDATA-2181] Thread Leak during compaction proc...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2086
retest this please
---
[GitHub] carbondata pull request #2086: [CARBONDATA-2181] Thread Leak during compacti...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/carbondata/pull/2086
---
[GitHub] carbondata issue #2086: [CARBONDATA-2181] Thread Leak during compaction proc...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2086
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3992/
---
[GitHub] carbondata issue #2086: [CARBONDATA-2181] Thread Leak during compaction proc...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2086
LGTM
---
[GitHub] carbondata issue #2086: [CARBONDATA-2181] Thread Leak during compaction proc...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2086
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4497/
---
[GitHub] carbondata issue #2086: [CARBONDATA-2181] Thread Leak during compaction proc...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2086
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4477/
---