You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by BJangir <gi...@git.apache.org> on 2018/07/15 16:25:05 UTC
[GitHub] carbondata pull request #2508: [CARBONDATA-2744]Streaming lock is not releas...
GitHub user BJangir opened a pull request:
https://github.com/apache/carbondata/pull/2508
[CARBONDATA-2744]Streaming lock is not released even Batch processing…
Issue Detail :- if Streaming Application is running , DDLs like finish streaming ,close streaming are blocked.
ideally DDLs like finish streaming ,close streaming should be blocked if Batch Processing is running. if Batch processing is not happening then DDL's should be allowed from JDBCServer/Beeline.
Root Cause :- Streaming lock is taken on application start and it is released onQueryTerminate event of CarbonStreamingQueryListener ,this event will be called when stop() is called on StreamingQuery which means streaming Lock will be released on Either Streaming Query should be terminated Or complete Streaming Application should be terminated till then all stream lock DDL's are blocked.
Solution :- on AddBatch take streaming lock and once Batch is processed ,release the streaming lock.
Note:-
a. If close streaming is called OR streaming Table is updated with 'streaming'='false' and on Trigger time to AddBatch, addBatch will throw Exception and StreamingQuery should be start again.
b. if DDLs like finish streaming ,close streaming started 1st and addBatch started 2nd. addBatch will throw "can not acquire lock" Exception and StreamingQuery should be start again.
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:
- [ ] Any interfaces changed?
NO
- [ ] Any backward compatibility impacted?
NO
- [ ] Document update required?
NO
- [ ] Testing done
Manually verified below scenarios
a. Call StreamingFinished when addBatch is Done.
b. Whether New Batch works after Streaming Finish DDL success. it creates new Streaming Segment
c. New Batch while Streaming Finnish DDL was running
d. Call Streaming Finish when Add Batch was Running
Verified above scenarios for other DDL' also (close streaming,set SET TBLPROPERTIES('streaming'='false')
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
NO
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/BJangir/incubator-carbondata streaming_lock
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/2508.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2508
----
commit 2ff88bad23a74ea9b0167574955c4e14c65ca755
Author: BJangir <ba...@...>
Date: 2018-07-15T16:14:02Z
[CARBONDATA-2744]Streaming lock is not released even Batch processing is not happening
----
---
[GitHub] carbondata pull request #2508: [CARBONDATA-2744]Streaming lock is not releas...
Posted by zzcclp <gi...@git.apache.org>.
Github user zzcclp commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2508#discussion_r202565677
--- Diff: integration/spark-common/src/main/scala/org/apache/carbondata/streaming/StreamSinkFactory.scala ---
@@ -78,13 +78,13 @@ object StreamSinkFactory {
}
}
+
--- End diff --
remove this blank line.
---
[GitHub] carbondata issue #2508: [CARBONDATA-2744]Streaming lock is not released even...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2508
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/39/
---
[GitHub] carbondata pull request #2508: [CARBONDATA-2744]Streaming lock is not releas...
Posted by zzcclp <gi...@git.apache.org>.
Github user zzcclp commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2508#discussion_r202566105
--- Diff: integration/spark-common/src/main/scala/org/apache/carbondata/streaming/StreamSinkFactory.scala ---
@@ -78,13 +78,13 @@ object StreamSinkFactory {
}
}
+
def createStreamTableSink(
sparkSession: SparkSession,
hadoopConf: Configuration,
carbonTable: CarbonTable,
parameters: Map[String, String]): Sink = {
--- End diff --
remove blank line
---
[GitHub] carbondata issue #2508: [CARBONDATA-2744]Streaming lock is not released even...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2508
Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2169/
---
[GitHub] carbondata issue #2508: [CARBONDATA-2744]Streaming lock is not released even...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2508
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1992/
---
[GitHub] carbondata issue #2508: [CARBONDATA-2744]Streaming lock is not released even...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2508
Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8788/
---
[GitHub] carbondata issue #2508: [CARBONDATA-2744]Streaming lock is not released even...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2508
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/543/
---
[GitHub] carbondata issue #2508: [CARBONDATA-2744]Streaming lock is not released even...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2508
Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/721/
---
[GitHub] carbondata issue #2508: [CARBONDATA-2744]Streaming lock is not released even...
Posted by QiangCai <gi...@git.apache.org>.
Github user QiangCai commented on the issue:
https://github.com/apache/carbondata/pull/2508
@BJangir
In my opinion, DDLs(finish streaming and close streaming) should be blocked if Stream is running.
---
[GitHub] carbondata issue #2508: [CARBONDATA-2744]Streaming lock is not released even...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2508
Build Failed with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10244/
---
[GitHub] carbondata issue #2508: [CARBONDATA-2744]Streaming lock is not released even...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2508
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7198/
---
[GitHub] carbondata issue #2508: [CARBONDATA-2744]Streaming lock is not released even...
Posted by zzcclp <gi...@git.apache.org>.
Github user zzcclp commented on the issue:
https://github.com/apache/carbondata/pull/2508
@BJangir I have one question: If close streaming is called OR streaming Table is updated with 'streaming'='false', it doesn't check 'streaming' property again in method 'addBatch', how to throw an exception?
The 'streaming' property is just checked before create 'CarbonAppendableStreamSink' in CarbonSource.
---
[GitHub] carbondata issue #2508: [CARBONDATA-2744]Streaming lock is not released even...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2508
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5972/
---
[GitHub] carbondata issue #2508: [CARBONDATA-2744]Streaming lock is not released even...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2508
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5864/
---