You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by xuchuanyin <gi...@git.apache.org> on 2018/06/26 06:21:41 UTC
[GitHub] carbondata pull request #2411: [CARBONDATA-2654][Datamap] Optimize output fo...
GitHub user xuchuanyin opened a pull request:
https://github.com/apache/carbondata/pull/2411
[CARBONDATA-2654][Datamap] Optimize output for explaining querying with datamap
Currently if we have multiple datamaps and the query hits all the
datamaps, carbondata explain command will only show the first datamap
and all the datamaps are not shown. In this commit, we show all the
datamaps that are hitted in this query.
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:
- [ ] Any interfaces changed?
`Yes, only internal interfaces have been changed`
- [ ] Any backward compatibility impacted?
`NO`
- [ ] Document update required?
`NO`
- [ ] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
`Tests added and updated`
- How it is tested? Please attach test report.
`Tested in local machine`
- Is it a performance related change? Please attach the performance test report.
`No`
- Any additional information to help reviewers in testing this change.
`NA`
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
`NA`
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/xuchuanyin/carbondata issue_2654_optimize_explain_dm
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/2411.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2411
----
commit 0cf11cd443cd4531180e0e674cc870dbed867fe3
Author: xuchuanyin <xu...@...>
Date: 2018-06-24T02:06:54Z
Fix bugs in bloomfilter for dictionary/sort/date column
In carbondata,
for dictionary column, carbon convert literal value to dict value, then
convert dict value to mdk value, at last it stores the mdk value as
internal value in carbonfile.
for other columns, carbon convert literal value to internal value using
field-converter.
Since bloomfilter datamap stores the internal value, during query we
should convert the literal value in filter to internal value in order to
match the value stored in bloomfilter datamap.
Changes are made:
1. FieldConverters were refactored to extract common value convert methods.
2. BloomQueryModel was optimized to support converting literal value to
internal value.
3. fix bugs for int/float/date/timestamp as bloom index column
4. fix bugs in dictionary/sort column as bloom index column
5. add tests
6. block (deferred) rebuild for bloom datamap (contains bugs that does
not fix in this commit)
commit 29640866150ccdd66f145259dfc19fd0d7672254
Author: xuchuanyin <xu...@...>
Date: 2018-06-25T06:02:40Z
fix bugs
commit ee9359f3ac336b18658beef961ca9e47e7b0d56c
Author: xuchuanyin <xu...@...>
Date: 2018-06-25T06:17:49Z
reduce data size for test
commit b910c94d924ac015bbcdae82e60808e2f41d0b46
Author: xuchuanyin <xu...@...>
Date: 2018-06-26T06:11:17Z
Fix bugs in negative number of skipped blocklets
Currently in carbondata, default blocklet datamap will be used to prune
blocklets. Then other indexdatamap will be used.
But the other index datamap works for segment scope, which in some
scenarios, the size of pruned result will be bigger than that of default
datamap, thus causing negative number of skipped blocklets in explain
query output.
Here we add intersection after pruning. If the pruned result size is
zero, we will finish the pruning.
commit e3944138c3a5c9d914369f9ff4766cbdc1afede6
Author: xuchuanyin <xu...@...>
Date: 2018-06-26T06:13:59Z
Optimize output for explaining querying with datamap
Currently if we have multiple datamaps and the query hits all the
datamaps, carbondata explain command will only show the first datamap
and all the datamaps are not shown. In this commit, we show all the
datamaps that are hitted in this query.
----
---
[GitHub] carbondata issue #2411: [CARBONDATA-2654][Datamap] Optimize output for expla...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2411
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5595/
---
[GitHub] carbondata pull request #2411: [CARBONDATA-2654][Datamap] Optimize output fo...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/carbondata/pull/2411
---
[GitHub] carbondata issue #2411: [CARBONDATA-2654][Datamap] Optimize output for expla...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on the issue:
https://github.com/apache/carbondata/pull/2411
retest this please
---
[GitHub] carbondata issue #2411: [CARBONDATA-2654][Datamap] Optimize output for expla...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2411
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5378/
---
[GitHub] carbondata issue #2411: [CARBONDATA-2654][Datamap] Optimize output for expla...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2411
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6549/
---
[GitHub] carbondata issue #2411: [CARBONDATA-2654][Datamap] Optimize output for expla...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on the issue:
https://github.com/apache/carbondata/pull/2411
retest this please
---
[GitHub] carbondata issue #2411: [CARBONDATA-2654][Datamap] Optimize output for expla...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2411
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6636/
---
[GitHub] carbondata issue #2411: [CARBONDATA-2654][Datamap] Optimize output for expla...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on the issue:
https://github.com/apache/carbondata/pull/2411
retest this please
---
[GitHub] carbondata issue #2411: [CARBONDATA-2654][Datamap] Optimize output for expla...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2411
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6572/
---
[GitHub] carbondata pull request #2411: [CARBONDATA-2654][Datamap] Optimize output fo...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2411#discussion_r200021222
--- Diff: core/src/main/java/org/apache/carbondata/core/datamap/dev/expr/DataMapWrapperSchema.java ---
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.core.datamap.dev.expr;
+
+import org.apache.carbondata.core.metadata.schema.table.DataMapSchema;
+
+/**
+ * schema for datamap wrapper
--- End diff --
OK, will change the name to 'DataMapWrapperSimpleInfo'
---
[GitHub] carbondata issue #2411: [CARBONDATA-2654][Datamap] Optimize output for expla...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2411
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6756/
---
[GitHub] carbondata issue #2411: [CARBONDATA-2654][Datamap] Optimize output for expla...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2411
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5588/
---
[GitHub] carbondata issue #2411: [CARBONDATA-2654][Datamap] Optimize output for expla...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2411
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5611/
---
[GitHub] carbondata issue #2411: [CARBONDATA-2654][Datamap] Optimize output for expla...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2411
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6545/
---
[GitHub] carbondata issue #2411: [CARBONDATA-2654][Datamap] Optimize output for expla...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on the issue:
https://github.com/apache/carbondata/pull/2411
LGTM
---
[GitHub] carbondata issue #2411: [CARBONDATA-2654][Datamap] Optimize output for expla...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2411
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5509/
---
[GitHub] carbondata issue #2411: [CARBONDATA-2654][Datamap] Optimize output for expla...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2411
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5453/
---
[GitHub] carbondata issue #2411: [CARBONDATA-2654][Datamap] Optimize output for expla...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2411
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5466/
---
[GitHub] carbondata pull request #2411: [CARBONDATA-2654][Datamap] Optimize output fo...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2411#discussion_r200017553
--- Diff: core/src/main/java/org/apache/carbondata/core/datamap/dev/expr/DataMapWrapperSchema.java ---
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.core.datamap.dev.expr;
+
+import org.apache.carbondata.core.metadata.schema.table.DataMapSchema;
+
+/**
+ * schema for datamap wrapper
--- End diff --
Can you describe why this is needed?
---
[GitHub] carbondata issue #2411: [CARBONDATA-2654][Datamap] Optimize output for expla...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2411
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5450/
---
[GitHub] carbondata pull request #2411: [CARBONDATA-2654][Datamap] Optimize output fo...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2411#discussion_r200018301
--- Diff: core/src/main/java/org/apache/carbondata/core/datamap/dev/expr/DataMapWrapperSchema.java ---
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.core.datamap.dev.expr;
+
+import org.apache.carbondata.core.metadata.schema.table.DataMapSchema;
+
+/**
+ * schema for datamap wrapper
--- End diff --
Because currently a DatamapWrapper contains more than one datamap, this class is to describle its schema.
For example a AndDatamapWrapper contains BloomFilterin its left and Lucene in its right, then its schema would be AND(BloomFilter, Lucene).
This DataMapWrapperSchema is used for explain output
---
[GitHub] carbondata issue #2411: [CARBONDATA-2654][Datamap] Optimize output for expla...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2411
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5504/
---
[GitHub] carbondata issue #2411: [CARBONDATA-2654][Datamap] Optimize output for expla...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2411
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5377/
---
[GitHub] carbondata issue #2411: [CARBONDATA-2654][Datamap] Optimize output for expla...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2411
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5401/
---
[GitHub] carbondata issue #2411: [CARBONDATA-2654][Datamap] Optimize output for expla...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2411
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6547/
---
[GitHub] carbondata issue #2411: [CARBONDATA-2654][Datamap] Optimize output for expla...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2411
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5391/
---
[GitHub] carbondata pull request #2411: [CARBONDATA-2654][Datamap] Optimize output fo...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2411#discussion_r200018398
--- Diff: core/src/main/java/org/apache/carbondata/core/datamap/dev/expr/DataMapWrapperSchema.java ---
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.core.datamap.dev.expr;
+
+import org.apache.carbondata.core.metadata.schema.table.DataMapSchema;
+
+/**
+ * schema for datamap wrapper
+ */
+public class DataMapWrapperSchema {
+ enum WrapperType {
+ PRIMITIVE,
+ AND,
+ OR
+ }
+
+ private WrapperType wrapperType;
+ private DataMapWrapperSchema left;
+ private DataMapWrapperSchema right;
+ private DataMapSchema schema;
+
+ private DataMapWrapperSchema(WrapperType wrapperType, DataMapWrapperSchema left,
+ DataMapWrapperSchema right) {
+ this.wrapperType = wrapperType;
+ this.left = left;
+ this.right = right;
+ }
+
+ private DataMapWrapperSchema(DataMapSchema schema) {
+ this.wrapperType = WrapperType.PRIMITIVE;
+ this.schema = schema;
+ }
+
+ public static DataMapWrapperSchema fromDataMapWrapper(DataMapExprWrapper dataMapExprWrapper) {
+ if (dataMapExprWrapper instanceof DataMapExprWrapperImpl) {
+ return new DataMapWrapperSchema(
+ ((DataMapExprWrapperImpl) dataMapExprWrapper).getDataMapSchema());
+ } else if (dataMapExprWrapper instanceof AndDataMapExprWrapper) {
+ return new DataMapWrapperSchema(WrapperType.AND,
+ fromDataMapWrapper(dataMapExprWrapper.getLeftDataMapWrapper()),
+ fromDataMapWrapper(dataMapExprWrapper.getRightDataMapWrapprt()));
+ } else {
+ return new DataMapWrapperSchema(WrapperType.OR,
+ fromDataMapWrapper(dataMapExprWrapper.getLeftDataMapWrapper()),
+ fromDataMapWrapper(dataMapExprWrapper.getRightDataMapWrapprt()));
+ }
+ }
+
+ public String getDataMapWrapperName() {
+ if (WrapperType.PRIMITIVE == wrapperType) {
+ return schema.getDataMapName();
+ } else {
+ return String.format("%s(%s, %s)",
--- End diff --
FYI: This is the wrapperName
---
[GitHub] carbondata issue #2411: [CARBONDATA-2654][Datamap] Optimize output for expla...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2411
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6638/
---
[GitHub] carbondata pull request #2411: [CARBONDATA-2654][Datamap] Optimize output fo...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2411#discussion_r200018803
--- Diff: core/src/main/java/org/apache/carbondata/core/datamap/dev/expr/DataMapWrapperSchema.java ---
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.core.datamap.dev.expr;
+
+import org.apache.carbondata.core.metadata.schema.table.DataMapSchema;
+
+/**
+ * schema for datamap wrapper
--- End diff --
Can you add these in comment.
And if this class is only used for Explain, can you give a better name for it.
---
[GitHub] carbondata issue #2411: [CARBONDATA-2654][Datamap] Optimize output for expla...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2411
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5534/
---
[GitHub] carbondata issue #2411: [CARBONDATA-2654][Datamap] Optimize output for expla...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on the issue:
https://github.com/apache/carbondata/pull/2411
retest this please
---
[GitHub] carbondata issue #2411: [CARBONDATA-2654][Datamap] Optimize output for expla...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2411
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5452/
---
[GitHub] carbondata pull request #2411: [CARBONDATA-2654][Datamap] Optimize output fo...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2411#discussion_r200018440
--- Diff: core/src/main/java/org/apache/carbondata/core/datamap/dev/expr/DataMapWrapperSchema.java ---
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.core.datamap.dev.expr;
+
+import org.apache.carbondata.core.metadata.schema.table.DataMapSchema;
+
+/**
+ * schema for datamap wrapper
+ */
+public class DataMapWrapperSchema {
+ enum WrapperType {
+ PRIMITIVE,
+ AND,
+ OR
+ }
+
+ private WrapperType wrapperType;
+ private DataMapWrapperSchema left;
+ private DataMapWrapperSchema right;
+ private DataMapSchema schema;
+
+ private DataMapWrapperSchema(WrapperType wrapperType, DataMapWrapperSchema left,
+ DataMapWrapperSchema right) {
+ this.wrapperType = wrapperType;
+ this.left = left;
+ this.right = right;
+ }
+
+ private DataMapWrapperSchema(DataMapSchema schema) {
+ this.wrapperType = WrapperType.PRIMITIVE;
+ this.schema = schema;
+ }
+
+ public static DataMapWrapperSchema fromDataMapWrapper(DataMapExprWrapper dataMapExprWrapper) {
+ if (dataMapExprWrapper instanceof DataMapExprWrapperImpl) {
+ return new DataMapWrapperSchema(
+ ((DataMapExprWrapperImpl) dataMapExprWrapper).getDataMapSchema());
+ } else if (dataMapExprWrapper instanceof AndDataMapExprWrapper) {
+ return new DataMapWrapperSchema(WrapperType.AND,
+ fromDataMapWrapper(dataMapExprWrapper.getLeftDataMapWrapper()),
+ fromDataMapWrapper(dataMapExprWrapper.getRightDataMapWrapprt()));
+ } else {
+ return new DataMapWrapperSchema(WrapperType.OR,
+ fromDataMapWrapper(dataMapExprWrapper.getLeftDataMapWrapper()),
+ fromDataMapWrapper(dataMapExprWrapper.getRightDataMapWrapprt()));
+ }
+ }
+
+ public String getDataMapWrapperName() {
+ if (WrapperType.PRIMITIVE == wrapperType) {
+ return schema.getDataMapName();
+ } else {
+ return String.format("%s(%s, %s)",
+ wrapperType, left.getDataMapWrapperName(), right.getDataMapWrapperName());
+ }
+ }
+
+ public String getDataMapWrapperProvider() {
+ if (WrapperType.PRIMITIVE == wrapperType) {
+ return schema.getProviderName();
+ } else {
+ return String.format("%s(%s, %s)",
--- End diff --
FYI: This is the wrapperProvider
---
[GitHub] carbondata issue #2411: [CARBONDATA-2654][Datamap] Optimize output for expla...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2411
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5374/
---
[GitHub] carbondata issue #2411: [CARBONDATA-2654][Datamap] Optimize output for expla...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on the issue:
https://github.com/apache/carbondata/pull/2411
The output for explain looks like below:
```
Table Scan on carbon_bloom
- total blocklets: 1
- filter: (((((name <> null and city <> null) and id <> null) and id = 1) and city = city_1) and name = n1)
- pruned by Main DataMap
- skipped blocklets: 0
- pruned by CG DataMap
- name: AND(AND(datamap11, datamap13), datamap12)
- provider: AND(AND(bloomfilter, bloomfilter), bloomfilter)
- skipped blocklets: 0
```
Before this PR, the output only shows `datamap11`
---
[GitHub] carbondata issue #2411: [CARBONDATA-2654][Datamap] Optimize output for expla...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2411
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6559/
---
[GitHub] carbondata issue #2411: [CARBONDATA-2654][Datamap] Optimize output for expla...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2411
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6679/
---
[GitHub] carbondata issue #2411: [CARBONDATA-2654][Datamap] Optimize output for expla...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on the issue:
https://github.com/apache/carbondata/pull/2411
This PR only contains commit `Optimize output for explaining querying with datamap`
It depends on PR #2403 and PR #2410
---