You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by ajantha-bhat <gi...@git.apache.org> on 2018/04/30 06:16:54 UTC
[GitHub] carbondata pull request #2251: [CARBONDATA-2417] SDK writer goes to infinite...
GitHub user ajantha-bhat opened a pull request:
https://github.com/apache/carbondata/pull/2251
[CARBONDATA-2417] SDK writer goes to infinite wait when consumer thread is dead
[CARBONDATA-2417]
problem: SDK writer goes to infinite wait when consumer thread is dead
root cause: due to bad record when exception happens at consumer thread
during write, this message is not reached producer (sdk writer).
So, SDK keeps writing data assuming consumer will consume it. But as
consumer is dead. Queue becomes full and queue.put() will be blocked
forever.
Solution: If cannot be added to queue, check whether consumer is alive
or not after every 10 seconds.
If not alive throw exception, if alive try again
- [ ] Any interfaces changed? no
- [ ] Any backward compatibility impacted? no
- [ ] Document update required? no
- [ ] Testing done
updated the testcase in TestNonTransactionalCarbonTable.scala
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. NA
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ajantha-bhat/carbondata branch3
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/2251.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2251
----
commit 7f19372fc32a6978ac5aa87f20f346e75a9b1e9d
Author: ajantha-bhat <aj...@...>
Date: 2018-04-28T12:47:35Z
[CARBONDATA-2417]
problem: SDK writer goes to infinite wait when cosumer thread is dead
root cause: due to bad record when exception happens at consumer thread
during write, this message is not reached producer (sdk writer).
So, SDK keeps writing data assuming consumer will consume it. But as
consumer is dead. Queue becomes full and queue.put() will be blocked
forever.
Solution: If cannot be added to queue, check whether consumer is alive
or not after every 10 seconds.
If not alive throw exception, if alive try again
----
---
[GitHub] carbondata issue #2251: [CARBONDATA-2417] SDK writer goes to infinite wait w...
Posted by ajantha-bhat <gi...@git.apache.org>.
Github user ajantha-bhat commented on the issue:
https://github.com/apache/carbondata/pull/2251
retest this please
---
[GitHub] carbondata issue #2251: [CARBONDATA-2417] SDK writer goes to infinite wait w...
Posted by ajantha-bhat <gi...@git.apache.org>.
Github user ajantha-bhat commented on the issue:
https://github.com/apache/carbondata/pull/2251
@ravipesala : Agree, But CarbonOutputIteratorWrapper.closeWriter will try to put batch to queue. so before putting to queue must check whether consumer is dead or busy. so above code changes are required
---
[GitHub] carbondata issue #2251: [CARBONDATA-2417] SDK writer goes to infinite wait w...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2251
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4646/
---
[GitHub] carbondata pull request #2251: [CARBONDATA-2417] SDK writer goes to infinite...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/carbondata/pull/2251
---
[GitHub] carbondata issue #2251: [CARBONDATA-2417] SDK writer goes to infinite wait w...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2251
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4647/
---
[GitHub] carbondata issue #2251: [CARBONDATA-2417] SDK writer goes to infinite wait w...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2251
LGTM
---
[GitHub] carbondata issue #2251: [CARBONDATA-2417] SDK writer goes to infinite wait w...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2251
@ajantha-bhat During any abnormalities it needs to call `CarbonOutputIteratorWrapper.closeWriter` it can close gracefully otherwise unexpected issues like this will come
---
[GitHub] carbondata issue #2251: [CARBONDATA-2417] SDK writer goes to infinite wait w...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2251
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5528/
---
[GitHub] carbondata issue #2251: [CARBONDATA-2417] SDK writer goes to infinite wait w...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2251
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4364/
---
[GitHub] carbondata issue #2251: [CARBONDATA-2417] SDK writer goes to infinite wait w...
Posted by ajantha-bhat <gi...@git.apache.org>.
Github user ajantha-bhat commented on the issue:
https://github.com/apache/carbondata/pull/2251
@ravipesala : handled as per comments.
---
[GitHub] carbondata issue #2251: [CARBONDATA-2417] SDK writer goes to infinite wait w...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2251
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5552/
---
[GitHub] carbondata issue #2251: [CARBONDATA-2417] SDK writer goes to infinite wait w...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2251
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4389/
---
[GitHub] carbondata issue #2251: [CARBONDATA-2417] SDK writer goes to infinite wait w...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2251
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4648/
---
[GitHub] carbondata issue #2251: [CARBONDATA-2417] SDK writer goes to infinite wait w...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2251
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4630/
---