You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by QiangCai <gi...@git.apache.org> on 2016/12/06 13:34:47 UTC
[GitHub] incubator-carbondata pull request #403: [CARBONDATA-497][SPARK2]fix datatype...
GitHub user QiangCai opened a pull request:
https://github.com/apache/incubator-carbondata/pull/403
[CARBONDATA-497][SPARK2]fix datatype issue of CarbonLateDecoderRule
1. Fix the data type of dictionary dimension to resolve the logical plan
2. Perfect translateFilter method to push down more filters to CarbonScanRDD.
3. Add decimal type field to CarbonExample
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/QiangCai/incubator-carbondata fixbugforlatedecoder
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-carbondata/pull/403.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #403
----
commit 7159713725ac6bef057e27144021cdd06e4adba0
Author: QiangCai <qi...@qq.com>
Date: 2016-12-06T09:40:21Z
fixlatedecoder
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #403: [CARBONDATA-497][SPARK2]fix datatype issue ...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/incubator-carbondata/pull/403
Build Success with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/58/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #403: [CARBONDATA-497][SPARK2]fix datatype issue ...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/incubator-carbondata/pull/403
Build Success, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/44/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #403: [CARBONDATA-497][SPARK2]fix datatype issue ...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/incubator-carbondata/pull/403
Build Success, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/39/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #403: [CARBONDATA-497][SPARK2]fix datatype issue ...
Posted by QiangCai <gi...@git.apache.org>.
Github user QiangCai commented on the issue:
https://github.com/apache/incubator-carbondata/pull/403
OK to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #403: [CARBONDATA-497][SPARK2]fix datatype...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-carbondata/pull/403
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #403: [CARBONDATA-497][SPARK2]fix datatype issue ...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on the issue:
https://github.com/apache/incubator-carbondata/pull/403
ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #403: [CARBONDATA-497][SPARK2]fix datatype issue ...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/incubator-carbondata/pull/403
Build Success with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/60/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #403: [CARBONDATA-497][SPARK2]fix datatype issue ...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/incubator-carbondata/pull/403
Build Success, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/38/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #403: [CARBONDATA-497][SPARK2]fix datatype issue ...
Posted by QiangCai <gi...@git.apache.org>.
Github user QiangCai commented on the issue:
https://github.com/apache/incubator-carbondata/pull/403
test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #403: [CARBONDATA-497][SPARK2]fix datatype issue ...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on the issue:
https://github.com/apache/incubator-carbondata/pull/403
test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #403: [CARBONDATA-497][SPARK2]fix datatype...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/403#discussion_r91113306
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/optimizer/CarbonLateDecodeRule.scala ---
@@ -456,76 +487,143 @@ class CarbonLateDecodeRule extends Rule[LogicalPlan] with PredicateHelper {
case cd: CarbonDictionaryCatalystDecoder =>
cd
case sort: Sort =>
+ val tmpAttrMap = new mutable.HashMap[AttributeReferenceWrapper, Attribute]()
+ if (sort.child.isInstanceOf[CarbonDictionaryTempDecoder]) {
+ val tempDecoder = sort.child.asInstanceOf[CarbonDictionaryTempDecoder]
+ tempDecoder.attrList.asScala.foreach{attr => tmpAttrMap.put(attr, attr.attr)}
+ }
val sortExprs = sort.order.map { s =>
s.transform {
case attr: AttributeReference =>
- updateDataType(attr, attrMap, allAttrsNotDecode, aliasMap)
+ val tempAttr = tmpAttrMap.get(AttributeReferenceWrapper(attr))
+ if(tempAttr.isDefined) {
+ tempAttr.get
+ } else {
+ updateDataType(attr, attrMap, allAttrsNotDecode, aliasMap)
+ }
}.asInstanceOf[SortOrder]
}
Sort(sortExprs, sort.global, sort.child)
case agg: Aggregate if !agg.child.isInstanceOf[CarbonDictionaryCatalystDecoder] =>
+ val tmpAttrMap = new mutable.HashMap[AttributeReferenceWrapper, Attribute]()
+ if (agg.child.isInstanceOf[CarbonDictionaryTempDecoder]) {
+ val tempDecoder = agg.child.asInstanceOf[CarbonDictionaryTempDecoder]
+ tempDecoder.attrList.asScala.foreach{attr => tmpAttrMap.put(attr, attr.attr)}
+ }
+
val aggExps = agg.aggregateExpressions.map { aggExp =>
aggExp.transform {
case attr: AttributeReference =>
- updateDataType(attr, attrMap, allAttrsNotDecode, aliasMap)
+ val tempAttr = tmpAttrMap.get(AttributeReferenceWrapper(attr))
+ if(tempAttr.isDefined) {
+ tempAttr.get
+ } else {
+ updateDataType(attr, attrMap, allAttrsNotDecode, aliasMap)
+ }
}
}.asInstanceOf[Seq[NamedExpression]]
val grpExps = agg.groupingExpressions.map { gexp =>
gexp.transform {
case attr: AttributeReference =>
- updateDataType(attr, attrMap, allAttrsNotDecode, aliasMap)
+ val tempAttr = tmpAttrMap.get(AttributeReferenceWrapper(attr))
+ if(tempAttr.isDefined) {
+ tempAttr.get
+ } else {
+ updateDataType(attr, attrMap, allAttrsNotDecode, aliasMap)
+ }
}
}
Aggregate(grpExps, aggExps, agg.child)
case expand: Expand =>
+ val tmpAttrMap = new mutable.HashMap[AttributeReferenceWrapper, Attribute]()
+ if (expand.child.isInstanceOf[CarbonDictionaryTempDecoder]) {
+ val tempDecoder = expand.child.asInstanceOf[CarbonDictionaryTempDecoder]
+ tempDecoder.attrList.asScala.foreach{attr => tmpAttrMap.put(attr, attr.attr)}
+ }
expand.transformExpressions {
case attr: AttributeReference =>
- updateDataType(attr, attrMap, allAttrsNotDecode, aliasMap)
+ val tempAttr = tmpAttrMap.get(AttributeReferenceWrapper(attr))
+ if(tempAttr.isDefined) {
+ tempAttr.get
+ } else {
+ updateDataType(attr, attrMap, allAttrsNotDecode, aliasMap)
+ }
}
case filter: Filter =>
- val filterExps = filter.condition transform {
- case attr: AttributeReference =>
- updateDataType(attr, attrMap, allAttrsNotDecode, aliasMap)
- }
- Filter(filterExps, filter.child)
+ filter
case j: Join =>
marker.pushBinaryMarker(allAttrsNotDecode)
j
case u: Union =>
marker.pushBinaryMarker(allAttrsNotDecode)
u
case p: Project if relations.nonEmpty =>
+ val tmpAttrMap = new mutable.HashMap[AttributeReferenceWrapper, Attribute]()
+ if (p.child.isInstanceOf[CarbonDictionaryTempDecoder]) {
+ val tempDecoder = p.child.asInstanceOf[CarbonDictionaryTempDecoder]
+ tempDecoder.attrList.asScala.foreach{attr => tmpAttrMap.put(attr, attr.attr)}
+ }
val prExps = p.projectList.map { prExp =>
- prExp.transform {
- case attr: AttributeReference =>
- updateDataType(attr, attrMap, allAttrsNotDecode, aliasMap)
- }
+ prExp.transform {
--- End diff --
feels like incorrect indentation
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #403: [CARBONDATA-497][SPARK2]fix datatype...
Posted by QiangCai <gi...@git.apache.org>.
Github user QiangCai commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/403#discussion_r91222679
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/optimizer/CarbonLateDecodeRule.scala ---
@@ -456,76 +487,143 @@ class CarbonLateDecodeRule extends Rule[LogicalPlan] with PredicateHelper {
case cd: CarbonDictionaryCatalystDecoder =>
cd
case sort: Sort =>
+ val tmpAttrMap = new mutable.HashMap[AttributeReferenceWrapper, Attribute]()
+ if (sort.child.isInstanceOf[CarbonDictionaryTempDecoder]) {
+ val tempDecoder = sort.child.asInstanceOf[CarbonDictionaryTempDecoder]
+ tempDecoder.attrList.asScala.foreach{attr => tmpAttrMap.put(attr, attr.attr)}
+ }
val sortExprs = sort.order.map { s =>
s.transform {
case attr: AttributeReference =>
- updateDataType(attr, attrMap, allAttrsNotDecode, aliasMap)
+ val tempAttr = tmpAttrMap.get(AttributeReferenceWrapper(attr))
+ if(tempAttr.isDefined) {
+ tempAttr.get
+ } else {
+ updateDataType(attr, attrMap, allAttrsNotDecode, aliasMap)
+ }
}.asInstanceOf[SortOrder]
}
Sort(sortExprs, sort.global, sort.child)
case agg: Aggregate if !agg.child.isInstanceOf[CarbonDictionaryCatalystDecoder] =>
+ val tmpAttrMap = new mutable.HashMap[AttributeReferenceWrapper, Attribute]()
+ if (agg.child.isInstanceOf[CarbonDictionaryTempDecoder]) {
+ val tempDecoder = agg.child.asInstanceOf[CarbonDictionaryTempDecoder]
+ tempDecoder.attrList.asScala.foreach{attr => tmpAttrMap.put(attr, attr.attr)}
+ }
+
val aggExps = agg.aggregateExpressions.map { aggExp =>
aggExp.transform {
case attr: AttributeReference =>
- updateDataType(attr, attrMap, allAttrsNotDecode, aliasMap)
+ val tempAttr = tmpAttrMap.get(AttributeReferenceWrapper(attr))
+ if(tempAttr.isDefined) {
+ tempAttr.get
+ } else {
+ updateDataType(attr, attrMap, allAttrsNotDecode, aliasMap)
+ }
}
}.asInstanceOf[Seq[NamedExpression]]
val grpExps = agg.groupingExpressions.map { gexp =>
gexp.transform {
case attr: AttributeReference =>
- updateDataType(attr, attrMap, allAttrsNotDecode, aliasMap)
+ val tempAttr = tmpAttrMap.get(AttributeReferenceWrapper(attr))
+ if(tempAttr.isDefined) {
+ tempAttr.get
+ } else {
+ updateDataType(attr, attrMap, allAttrsNotDecode, aliasMap)
+ }
}
}
Aggregate(grpExps, aggExps, agg.child)
case expand: Expand =>
+ val tmpAttrMap = new mutable.HashMap[AttributeReferenceWrapper, Attribute]()
+ if (expand.child.isInstanceOf[CarbonDictionaryTempDecoder]) {
+ val tempDecoder = expand.child.asInstanceOf[CarbonDictionaryTempDecoder]
+ tempDecoder.attrList.asScala.foreach{attr => tmpAttrMap.put(attr, attr.attr)}
+ }
expand.transformExpressions {
case attr: AttributeReference =>
- updateDataType(attr, attrMap, allAttrsNotDecode, aliasMap)
+ val tempAttr = tmpAttrMap.get(AttributeReferenceWrapper(attr))
+ if(tempAttr.isDefined) {
+ tempAttr.get
+ } else {
+ updateDataType(attr, attrMap, allAttrsNotDecode, aliasMap)
+ }
}
case filter: Filter =>
- val filterExps = filter.condition transform {
- case attr: AttributeReference =>
- updateDataType(attr, attrMap, allAttrsNotDecode, aliasMap)
- }
- Filter(filterExps, filter.child)
+ filter
case j: Join =>
marker.pushBinaryMarker(allAttrsNotDecode)
j
case u: Union =>
marker.pushBinaryMarker(allAttrsNotDecode)
u
case p: Project if relations.nonEmpty =>
+ val tmpAttrMap = new mutable.HashMap[AttributeReferenceWrapper, Attribute]()
+ if (p.child.isInstanceOf[CarbonDictionaryTempDecoder]) {
+ val tempDecoder = p.child.asInstanceOf[CarbonDictionaryTempDecoder]
+ tempDecoder.attrList.asScala.foreach{attr => tmpAttrMap.put(attr, attr.attr)}
+ }
val prExps = p.projectList.map { prExp =>
- prExp.transform {
- case attr: AttributeReference =>
- updateDataType(attr, attrMap, allAttrsNotDecode, aliasMap)
- }
+ prExp.transform {
--- End diff --
fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #403: [CARBONDATA-497][SPARK2]fix datatype...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/403#discussion_r91114234
--- Diff: examples/spark2/src/main/scala/org/apache/carbondata/examples/CarbonExample.scala ---
@@ -105,13 +106,14 @@ object CarbonExample {
s"""
| INSERT INTO TABLE carbon_table
| SELECT shortField, intField, bigintField, doubleField, stringField,
- | from_unixtime(unix_timestamp(timestampField,'yyyy/M/dd')) timestampField
--- End diff --
Can you move some of these SQL in the test case
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #403: [CARBONDATA-497][SPARK2]fix datatype issue ...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on the issue:
https://github.com/apache/incubator-carbondata/pull/403
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #403: [CARBONDATA-497][SPARK2]fix datatype issue ...
Posted by chenliang613 <gi...@git.apache.org>.
Github user chenliang613 commented on the issue:
https://github.com/apache/incubator-carbondata/pull/403
add to whitelist
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #403: [CARBONDATA-497][SPARK2]fix datatype issue ...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/incubator-carbondata/pull/403
Build Success with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/59/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---