You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by xuchuanyin <gi...@git.apache.org> on 2018/06/28 08:33:25 UTC
[GitHub] carbondata pull request #2425: [CARBONDATA-2637][BloomDataMap] Fix bugs for ...
GitHub user xuchuanyin opened a pull request:
https://github.com/apache/carbondata/pull/2425
[CARBONDATA-2637][BloomDataMap] Fix bugs for deferred rebuild for bloomfilter datamap
Previously when we implement ISSUE-2633, deferred rebuild for bloom
datamap is disabled for bloomfilter datamap due to unhandled bugs.
In this commit, we fixed the bugs and bring this feature back.
Since bloomfilter datamap create index on the carbon native raw bytes, we have to convert original literal value to carbon native bytes both in loading and querying.
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:
- [ ] Any interfaces changed?
- [ ] Any backward compatibility impacted?
- [ ] Document update required?
- [ ] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
`Tests added for deferred rebuild in different scenarios`
- How it is tested? Please attach test report.
`Tested in local machine`
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/xuchuanyin/carbondata issue_2637_bloom_deferred_rebuild
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/2425.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2425
----
commit 0cf11cd443cd4531180e0e674cc870dbed867fe3
Author: xuchuanyin <xu...@...>
Date: 2018-06-24T02:06:54Z
Fix bugs in bloomfilter for dictionary/sort/date column
In carbondata,
for dictionary column, carbon convert literal value to dict value, then
convert dict value to mdk value, at last it stores the mdk value as
internal value in carbonfile.
for other columns, carbon convert literal value to internal value using
field-converter.
Since bloomfilter datamap stores the internal value, during query we
should convert the literal value in filter to internal value in order to
match the value stored in bloomfilter datamap.
Changes are made:
1. FieldConverters were refactored to extract common value convert methods.
2. BloomQueryModel was optimized to support converting literal value to
internal value.
3. fix bugs for int/float/date/timestamp as bloom index column
4. fix bugs in dictionary/sort column as bloom index column
5. add tests
6. block (deferred) rebuild for bloom datamap (contains bugs that does
not fix in this commit)
commit 29640866150ccdd66f145259dfc19fd0d7672254
Author: xuchuanyin <xu...@...>
Date: 2018-06-25T06:02:40Z
fix bugs
commit ee9359f3ac336b18658beef961ca9e47e7b0d56c
Author: xuchuanyin <xu...@...>
Date: 2018-06-25T06:17:49Z
reduce data size for test
commit b910c94d924ac015bbcdae82e60808e2f41d0b46
Author: xuchuanyin <xu...@...>
Date: 2018-06-26T06:11:17Z
Fix bugs in negative number of skipped blocklets
Currently in carbondata, default blocklet datamap will be used to prune
blocklets. Then other indexdatamap will be used.
But the other index datamap works for segment scope, which in some
scenarios, the size of pruned result will be bigger than that of default
datamap, thus causing negative number of skipped blocklets in explain
query output.
Here we add intersection after pruning. If the pruned result size is
zero, we will finish the pruning.
commit 1b09402af8af95227f9af59f3c39eec4b3c5d493
Author: xuchuanyin <xu...@...>
Date: 2018-06-26T06:13:59Z
Optimize output for explaining querying with datamap
Currently if we have multiple datamaps and the query hits all the
datamaps, carbondata explain command will only show the first datamap
and all the datamaps are not shown. In this commit, we show all the
datamaps that are hitted in this query.
commit 46f03851c28975c5366853de9f86d9cad8ff3a4f
Author: xuchuanyin <xu...@...>
Date: 2018-06-26T11:53:15Z
Fix bugs in loading and querying on bloom column
Fix bugs in loading and querying with empty values on bloom index
columns. Convert null values to corresponding values.
commit 61168b5aa6864608070cc748c6cc3623df2f4c5e
Author: xuchuanyin <xu...@...>
Date: 2018-06-27T02:20:25Z
Add tests for querying on longstring bloom index column
add test to verify that bloom filter datamap support longstring column
commit 1aa79e009275614633087e23570d69bd97b2b78c
Author: xuchuanyin <xu...@...>
Date: 2018-06-28T08:25:49Z
Fix bugs for deferred rebuild for bloomfilter datamap
Previously when we implement ISSUE-2633, deferred rebuild for bloom
datamap is disabled for bloomfilter datamap due to unhandled bugs.
In this commit, we fixed the bugs and bring this feature back.
----
---
[GitHub] carbondata issue #2425: [CARBONDATA-2637][BloomDataMap] Fix bugs for deferre...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2425
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5698/
---
[GitHub] carbondata issue #2425: [CARBONDATA-2637][BloomDataMap] Fix bugs for deferre...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2425
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5676/
---
[GitHub] carbondata issue #2425: [CARBONDATA-2637][BloomDataMap] Fix bugs for deferre...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2425
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5684/
---
[GitHub] carbondata pull request #2425: [CARBONDATA-2637][BloomDataMap] Fix bugs for ...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2425#discussion_r200895451
--- Diff: integration/spark2/src/main/scala/org/apache/carbondata/datamap/IndexDataMapRebuildRDD.scala ---
@@ -144,13 +148,113 @@ class OriginalReadSupport(dataTypes: Array[DataType]) extends CarbonReadSupport[
}
}
+/**
+ * This class will generate row value which is raw bytes for the dimensions.
+ */
+class RawBytesReadSupport(segmentProperties: SegmentProperties, indexColumns: Array[CarbonColumn])
--- End diff --
Add CARBONDATA-2709 to refactor this
---
[GitHub] carbondata issue #2425: [CARBONDATA-2637][BloomDataMap] Fix bugs for deferre...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2425
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6916/
---
[GitHub] carbondata issue #2425: [CARBONDATA-2637][BloomDataMap] Fix bugs for deferre...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on the issue:
https://github.com/apache/carbondata/pull/2425
LGTM
---
[GitHub] carbondata issue #2425: [CARBONDATA-2637][BloomDataMap] Fix bugs for deferre...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2425
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6764/
---
[GitHub] carbondata issue #2425: [CARBONDATA-2637][BloomDataMap] Fix bugs for deferre...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2425
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5588/
---
[GitHub] carbondata issue #2425: [CARBONDATA-2637][BloomDataMap] Fix bugs for deferre...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2425
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5683/
---
[GitHub] carbondata pull request #2425: [CARBONDATA-2637][BloomDataMap] Fix bugs for ...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2425#discussion_r200892053
--- Diff: integration/spark2/src/main/scala/org/apache/carbondata/datamap/IndexDataMapRebuildRDD.scala ---
@@ -144,13 +148,113 @@ class OriginalReadSupport(dataTypes: Array[DataType]) extends CarbonReadSupport[
}
}
+/**
+ * This class will generate row value which is raw bytes for the dimensions.
+ */
+class RawBytesReadSupport(segmentProperties: SegmentProperties, indexColumns: Array[CarbonColumn])
--- End diff --
ok. I think we can refactor it later
---
[GitHub] carbondata issue #2425: [CARBONDATA-2637][BloomDataMap] Fix bugs for deferre...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on the issue:
https://github.com/apache/carbondata/pull/2425
retest sdv please
---
[GitHub] carbondata pull request #2425: [CARBONDATA-2637][BloomDataMap] Fix bugs for ...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2425#discussion_r200805465
--- Diff: integration/spark2/src/main/scala/org/apache/carbondata/datamap/IndexDataMapRebuildRDD.scala ---
@@ -144,13 +148,113 @@ class OriginalReadSupport(dataTypes: Array[DataType]) extends CarbonReadSupport[
}
}
+/**
+ * This class will generate row value which is raw bytes for the dimensions.
+ */
+class RawBytesReadSupport(segmentProperties: SegmentProperties, indexColumns: Array[CarbonColumn])
--- End diff --
ehh, There is another ReadSupport called 'OriginalReadSupport' in this file which is similar with this 'RawBytesReadSupport'. Do we need to move it too?
Besides, This class is only used IndexRebuildDmRDD, I think it's better to keep them here.
---
[GitHub] carbondata issue #2425: [CARBONDATA-2637][BloomDataMap] Fix bugs for deferre...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2425
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6918/
---
[GitHub] carbondata issue #2425: [CARBONDATA-2637][BloomDataMap] Fix bugs for deferre...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2425
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6744/
---
[GitHub] carbondata issue #2425: [CARBONDATA-2637][BloomDataMap] Fix bugs for deferre...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2425
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5575/
---
[GitHub] carbondata issue #2425: [CARBONDATA-2637][BloomDataMap] Fix bugs for deferre...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on the issue:
https://github.com/apache/carbondata/pull/2425
Do not trigger build before PR2416 is merged
---
[GitHub] carbondata pull request #2425: [CARBONDATA-2637][BloomDataMap] Fix bugs for ...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2425#discussion_r200655564
--- Diff: integration/spark2/src/main/scala/org/apache/carbondata/datamap/IndexDataMapRebuildRDD.scala ---
@@ -144,13 +148,113 @@ class OriginalReadSupport(dataTypes: Array[DataType]) extends CarbonReadSupport[
}
}
+/**
+ * This class will generate row value which is raw bytes for the dimensions.
+ */
+class RawBytesReadSupport(segmentProperties: SegmentProperties, indexColumns: Array[CarbonColumn])
--- End diff --
please add this as Java code in carbon-core module
---
[GitHub] carbondata issue #2425: [CARBONDATA-2637][BloomDataMap] Fix bugs for deferre...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on the issue:
https://github.com/apache/carbondata/pull/2425
Do not trigger build before PR2416 is merged
---
[GitHub] carbondata issue #2425: [CARBONDATA-2637][BloomDataMap] Fix bugs for deferre...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on the issue:
https://github.com/apache/carbondata/pull/2425
This PR depends on PR #2416
---
[GitHub] carbondata issue #2425: [CARBONDATA-2637][BloomDataMap] Fix bugs for deferre...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2425
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5515/
---
[GitHub] carbondata pull request #2425: [CARBONDATA-2637][BloomDataMap] Fix bugs for ...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2425#discussion_r200813333
--- Diff: integration/spark2/src/main/scala/org/apache/carbondata/datamap/IndexDataMapRebuildRDD.scala ---
@@ -144,13 +148,113 @@ class OriginalReadSupport(dataTypes: Array[DataType]) extends CarbonReadSupport[
}
}
+/**
+ * This class will generate row value which is raw bytes for the dimensions.
+ */
+class RawBytesReadSupport(segmentProperties: SegmentProperties, indexColumns: Array[CarbonColumn])
--- End diff --
besides, currently I can not find a readsupport that in module carbon-core, so better to leave it where it is needed.
---
[GitHub] carbondata issue #2425: [CARBONDATA-2637][BloomDataMap] Fix bugs for deferre...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2425
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5607/
---
[GitHub] carbondata issue #2425: [CARBONDATA-2637][BloomDataMap] Fix bugs for deferre...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2425
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5703/
---
[GitHub] carbondata issue #2425: [CARBONDATA-2637][BloomDataMap] Fix bugs for deferre...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on the issue:
https://github.com/apache/carbondata/pull/2425
Do not trigger build before PR2413 is merged
---
[GitHub] carbondata issue #2425: [CARBONDATA-2637][BloomDataMap] Fix bugs for deferre...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2425
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6685/
---
[GitHub] carbondata issue #2425: [CARBONDATA-2637][BloomDataMap] Fix bugs for deferre...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2425
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5540/
---
[GitHub] carbondata issue #2425: [CARBONDATA-2637][BloomDataMap] Fix bugs for deferre...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2425
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5701/
---
[GitHub] carbondata pull request #2425: [CARBONDATA-2637][BloomDataMap] Fix bugs for ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/carbondata/pull/2425
---
[GitHub] carbondata issue #2425: [CARBONDATA-2637][BloomDataMap] Fix bugs for deferre...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on the issue:
https://github.com/apache/carbondata/pull/2425
retest sdv please
---
[GitHub] carbondata issue #2425: [CARBONDATA-2637][BloomDataMap] Fix bugs for deferre...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on the issue:
https://github.com/apache/carbondata/pull/2425
This PR depends on PR #2416
---
[GitHub] carbondata issue #2425: [CARBONDATA-2637][BloomDataMap] Fix bugs for deferre...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2425
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5710/
---
[GitHub] carbondata issue #2425: [CARBONDATA-2637][BloomDataMap] Fix bugs for deferre...
Posted by brijoobopanna <gi...@git.apache.org>.
Github user brijoobopanna commented on the issue:
https://github.com/apache/carbondata/pull/2425
retest sdv please
---
[GitHub] carbondata issue #2425: [CARBONDATA-2637][BloomDataMap] Fix bugs for deferre...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2425
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5704/
---
[GitHub] carbondata issue #2425: [CARBONDATA-2637][BloomDataMap] Fix bugs for deferre...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2425
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5696/
---
[GitHub] carbondata issue #2425: [CARBONDATA-2637][BloomDataMap] Fix bugs for deferre...
Posted by brijoobopanna <gi...@git.apache.org>.
Github user brijoobopanna commented on the issue:
https://github.com/apache/carbondata/pull/2425
retest sdv please
---
[GitHub] carbondata issue #2425: [CARBONDATA-2637][BloomDataMap] Fix bugs for deferre...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2425
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6913/
---