You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@carbondata.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2016/07/25 15:55:20 UTC
[jira] [Commented] (CARBONDATA-107) Remove unnecessary ConverToSafe
in spark planner
[ https://issues.apache.org/jira/browse/CARBONDATA-107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15392151#comment-15392151 ]
ASF GitHub Bot commented on CARBONDATA-107:
-------------------------------------------
GitHub user jackylk opened a pull request:
https://github.com/apache/incubator-carbondata/pull/54
[CARBONDATA-107] remove unnecessary ConvertToSafe
CarbonDictionaryDecoder is using InternalRow only, so it should be able to process UnsafeRow.
By changing `canProcessUnsafeRows` and `canProcessSafeRows` to `true`, the planner will remove unnecessary ConvertToSafe operator
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jackylk/incubator-carbondata unsafe
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-carbondata/pull/54.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #54
----
commit a5bd2412d7067bddb1f61c4796bc704729252eb6
Author: jackylk <ja...@huawei.com>
Date: 2016-07-25T15:49:14Z
remove unnecessary ConvertToSafe
----
> Remove unnecessary ConverToSafe in spark planner
> ------------------------------------------------
>
> Key: CARBONDATA-107
> URL: https://issues.apache.org/jira/browse/CARBONDATA-107
> Project: CarbonData
> Issue Type: Improvement
> Components: carbon-spark
> Affects Versions: Apache CarbonData 0.1.0-incubating
> Reporter: Jacky Li
> Fix For: Apache CarbonData 0.1.0-incubating
>
>
> Query:
> select ch, sum(c) from (select ch,count(1) as c from t2 group by ch) temp where c > 1 group by ch
> Output plan looks like:
> == Physical Plan ==
> Limit 21
> ConvertToSafe
> CarbonDictionaryDecoder [CarbonDecoderRelation(Map(word#39 -> word#39, ch#40 -> ch#40, value#41 -> value#41),CarbonDatasourceRelation(`default`.`t1`,None))], ExcludeProfile(ArrayBuffer(#103)), CarbonAliasDecoderRelation()
> ConvertToSafe
> TungstenAggregate(key=[ch#40], functions=[(sum(c#101L),mode=Final,isDistinct=false)], output=[ch#40,_c1#102L])
> TungstenAggregate(key=[ch#40], functions=[(sum(c#101L),mode=Partial,isDistinct=false)], output=[ch#40,currentSum#122L])
> Filter (c#101L > FakeCarbonCast(1 as bigint))
> CarbonDictionaryDecoder [CarbonDecoderRelation(Map(word#39 -> word#39, ch#40 -> ch#40, value#41 -> value#41),CarbonDatasourceRelation(`default`.`t1`,None))], IncludeProfile(ArrayBuffer(#103)), CarbonAliasDecoderRelation()
> ConvertToSafe
> TungstenAggregate(key=[ch#40], functions=[(count(1),mode=Final,isDistinct=false)], output=[ch#40,c#101L])
> TungstenExchange hashpartitioning(ch#40)
> TungstenAggregate(key=[ch#40], functions=[(count(1),mode=Partial,isDistinct=false)], output=[ch#40,currentCount#126L])
> Project [ch#40]
> ConvertToSafe
> CarbonScan [ch#40], (CarbonRelation default, t1, CarbonMetaData(ArrayBuffer(word, ch),ArrayBuffer(value),org.carbondata.core.carbon.metadata.schema.table.CarbonTable@52d54ca2,DictionaryMap(Map(word -> true, ch -> true))), TableMeta(default_t1,/Users/jackylk/code/incubator-carbondata/target/store,org.carbondata.core.carbon.metadata.schema.table.CarbonTable@52d54ca2,Partitioner(org.carbondata.spark.partition.api.impl.SampleDataPartitionerImpl,[Ljava.lang.String;@62a877f4,1,[Ljava.lang.String;@3c180da5)), None), true
> There are unnecessary ConvertToSafe before CarbonDictionaryDecoder.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)