You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by QiangCai <gi...@git.apache.org> on 2018/10/18 03:08:59 UTC
[GitHub] carbondata pull request #2832: [CARBONDATA-3021][Streaming] Fix unsupported ...
GitHub user QiangCai opened a pull request:
https://github.com/apache/carbondata/pull/2832
[CARBONDATA-3021][Streaming] Fix unsupported data type exception for streaming
Background:
when spark uses Kryo serialization, streaming app throws the exception "Unsupported data type".
Root cause:
1. collect the data type list to driver side from executor side.
2. when using Kryo, datatype single instances are not working.
Solution:
don't collect measure data type list from executor side to avoid serialization.
- [x] Any interfaces changed?
no
- [x] Any backward compatibility impacted?
no
- [x] Document update required?
no
- [x] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.
use old test case to test it.
- [x] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
small
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/QiangCai/carbondata streaming_measure_datatype
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/2832.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2832
----
commit 36d72749632cb872907476592e8e8126bd298579
Author: QiangCai <qi...@...>
Date: 2018-10-18T02:56:13Z
fix unsupported data type exception for streaming
----
---
[GitHub] carbondata issue #2832: [CARBONDATA-3021][Streaming] Fix unsupported data ty...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2832
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/856/
---
[GitHub] carbondata pull request #2832: [CARBONDATA-3021][Streaming] Fix unsupported ...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2832#discussion_r226211737
--- Diff: integration/spark-common/src/main/scala/org/apache/spark/sql/execution/streaming/CarbonAppendableStreamSink.scala ---
@@ -102,6 +102,16 @@ class CarbonAppendableStreamSink(
CarbonProperties.getInstance().isEnableAutoHandoff
)
+ // measure data type array
+ private lazy val msrDataTypes = {
+ val msrList = carbonLoadModel.getCarbonDataLoadSchema.getCarbonTable.getMeasures
--- End diff --
suggest to use map instead of `(0 until msrDataTypes.length).foreach`
---
[GitHub] carbondata issue #2832: [CARBONDATA-3021][Streaming] Fix unsupported data ty...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2832
Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1068/
---
[GitHub] carbondata pull request #2832: [CARBONDATA-3021][Streaming] Fix unsupported ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/carbondata/pull/2832
---
[GitHub] carbondata issue #2832: [CARBONDATA-3021][Streaming] Fix unsupported data ty...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2832
Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9119/
---
[GitHub] carbondata issue #2832: [CARBONDATA-3021][Streaming] Fix unsupported data ty...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2832
Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1051/
---
[GitHub] carbondata issue #2832: [CARBONDATA-3021][Streaming] Fix unsupported data ty...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on the issue:
https://github.com/apache/carbondata/pull/2832
LGTM
---
[GitHub] carbondata issue #2832: [CARBONDATA-3021][Streaming] Fix unsupported data ty...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2832
Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9123/
---
[GitHub] carbondata issue #2832: [CARBONDATA-3021][Streaming] Fix unsupported data ty...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2832
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/854/
---
[GitHub] carbondata pull request #2832: [CARBONDATA-3021][Streaming] Fix unsupported ...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2832#discussion_r226211539
--- Diff: integration/spark-common/src/main/scala/org/apache/spark/sql/execution/streaming/CarbonAppendableStreamSink.scala ---
@@ -102,6 +102,16 @@ class CarbonAppendableStreamSink(
CarbonProperties.getInstance().isEnableAutoHandoff
)
+ // measure data type array
+ private lazy val msrDataTypes = {
+ val msrList = carbonLoadModel.getCarbonDataLoadSchema.getCarbonTable.getMeasures
--- End diff --
```suggestion
val msrList = carbonLoadModel.getCarbonDataLoadSchema.getCarbonTable.getMeasures.asScala.map
```
---
[GitHub] carbondata issue #2832: [CARBONDATA-3021][Streaming] Fix unsupported data ty...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2832
Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1041/
---
[GitHub] carbondata issue #2832: [CARBONDATA-3021][Streaming] Fix unsupported data ty...
Posted by zzcclp <gi...@git.apache.org>.
Github user zzcclp commented on the issue:
https://github.com/apache/carbondata/pull/2832
LGTM
---
[GitHub] carbondata issue #2832: [CARBONDATA-3021][Streaming] Fix unsupported data ty...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2832
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/844/
---
[GitHub] carbondata issue #2832: [CARBONDATA-3021][Streaming] Fix unsupported data ty...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2832
Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9135/
---
[GitHub] carbondata issue #2832: [CARBONDATA-3021][Streaming] Fix unsupported data ty...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2832
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/869/
---
[GitHub] carbondata issue #2832: [CARBONDATA-3021][Streaming] Fix unsupported data ty...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2832
Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1056/
---
[GitHub] carbondata issue #2832: [CARBONDATA-3021][Streaming] Fix unsupported data ty...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2832
Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9109/
---
[GitHub] carbondata issue #2832: [CARBONDATA-3021][Streaming] Fix unsupported data ty...
Posted by zzcclp <gi...@git.apache.org>.
Github user zzcclp commented on the issue:
https://github.com/apache/carbondata/pull/2832
retest this please
---