You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by ravipesala <gi...@git.apache.org> on 2017/03/14 08:27:40 UTC
[GitHub] incubator-carbondata pull request #653: [CARBONDATA-769] Added codegen suppo...
GitHub user ravipesala opened a pull request:
https://github.com/apache/incubator-carbondata/pull/653
[CARBONDATA-769] Added codegen support to spark2.1 carbon dictionary decoder.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ravipesala/incubator-carbondata measure-filter
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-carbondata/pull/653.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #653
----
commit 91104245c482b93b8bdf90775a083a401544f26e
Author: ravipesala <ra...@gmail.com>
Date: 2017-03-14T07:44:38Z
Added codegen support to spark2.1 carbon decoder.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #653: [CARBONDATA-769] Added codegen suppo...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/653#discussion_r106162467
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/CarbonDictionaryDecoder.scala ---
@@ -200,6 +201,123 @@ case class CarbonDictionaryDecoder(
}
}
+ override def doConsume(ctx: CodegenContext, input: Seq[ExprCode], row: ExprCode): String = {
+
+ val storePath = CarbonEnv.get.carbonMetastore.storePath
+ val absoluteTableIdentifiers = relations.map { relation =>
+ val carbonTable = relation.carbonRelation.carbonRelation.metaData.carbonTable
+ (carbonTable.getFactTableName, carbonTable.getAbsoluteTableIdentifier)
+ }.toMap
+
+ if (isRequiredToDecode) {
+ val cacheProvider: CacheProvider = CacheProvider.getInstance
+ val forwardDictionaryCache: Cache[DictionaryColumnUniqueIdentifier, Dictionary] =
+ cacheProvider.createCache(CacheType.FORWARD_DICTIONARY, storePath)
+ val dicts: Seq[ForwardDictionaryWrapper] = getDictionaryWrapper(absoluteTableIdentifiers,
+ forwardDictionaryCache, storePath)
+
+ val exprs = child.output.map(x =>
+ ExpressionCanonicalizer.execute(BindReferences.bindReference(x, child.output)))
+ ctx.currentVars = input
+ val resultVars = exprs.zipWithIndex.map { e =>
--- End diff --
ok
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #653: [CARBONDATA-769] Added codegen suppo...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-carbondata/pull/653
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #653: [CARBONDATA-769] Added codegen suppo...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala closed the pull request at:
https://github.com/apache/incubator-carbondata/pull/653
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #653: [CARBONDATA-769] Added codegen suppo...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/653#discussion_r105921333
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/CarbonDictionaryDecoder.scala ---
@@ -200,6 +201,123 @@ case class CarbonDictionaryDecoder(
}
}
+ override def doConsume(ctx: CodegenContext, input: Seq[ExprCode], row: ExprCode): String = {
+
+ val storePath = CarbonEnv.get.carbonMetastore.storePath
+ val absoluteTableIdentifiers = relations.map { relation =>
+ val carbonTable = relation.carbonRelation.carbonRelation.metaData.carbonTable
+ (carbonTable.getFactTableName, carbonTable.getAbsoluteTableIdentifier)
+ }.toMap
+
+ if (isRequiredToDecode) {
+ val cacheProvider: CacheProvider = CacheProvider.getInstance
+ val forwardDictionaryCache: Cache[DictionaryColumnUniqueIdentifier, Dictionary] =
+ cacheProvider.createCache(CacheType.FORWARD_DICTIONARY, storePath)
+ val dicts: Seq[ForwardDictionaryWrapper] = getDictionaryWrapper(absoluteTableIdentifiers,
+ forwardDictionaryCache, storePath)
+
+ val exprs = child.output.map(x =>
--- End diff --
change `map(x` to `map { x`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #653: [CARBONDATA-769] Added codegen support to s...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/incubator-carbondata/pull/653
Build Success with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1127/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #653: [CARBONDATA-769] Added codegen suppo...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/653#discussion_r105920583
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/CarbonDictionaryDecoder.scala ---
@@ -200,6 +201,123 @@ case class CarbonDictionaryDecoder(
}
}
+ override def doConsume(ctx: CodegenContext, input: Seq[ExprCode], row: ExprCode): String = {
+
+ val storePath = CarbonEnv.get.carbonMetastore.storePath
+ val absoluteTableIdentifiers = relations.map { relation =>
+ val carbonTable = relation.carbonRelation.carbonRelation.metaData.carbonTable
+ (carbonTable.getFactTableName, carbonTable.getAbsoluteTableIdentifier)
+ }.toMap
+
+ if (isRequiredToDecode) {
+ val cacheProvider: CacheProvider = CacheProvider.getInstance
+ val forwardDictionaryCache: Cache[DictionaryColumnUniqueIdentifier, Dictionary] =
+ cacheProvider.createCache(CacheType.FORWARD_DICTIONARY, storePath)
+ val dicts: Seq[ForwardDictionaryWrapper] = getDictionaryWrapper(absoluteTableIdentifiers,
+ forwardDictionaryCache, storePath)
+
+ val exprs = child.output.map(x =>
+ ExpressionCanonicalizer.execute(BindReferences.bindReference(x, child.output)))
+ ctx.currentVars = input
+ val resultVars = exprs.zipWithIndex.map { e =>
+ if (dicts(e._2) != null) {
+ val ev = e._1.genCode(ctx)
+ val value = ctx.freshName("value")
+ val valueIntern = ctx.freshName("valueIntern")
+ val isNull = ctx.freshName("isNull")
+ val dictsRef = ctx.addReferenceObj("dictsRef", dicts(e._2))
+ var code =
+ s"""
+ |${ev.code}
+ """.stripMargin
+ code +=
+ s"""
+ |boolean $isNull = false;
+ |byte[] $valueIntern = $dictsRef.getDictionaryValueForKeyInBytes(${ ev.value });
+ |if (java.util.Arrays.equals(org.apache.carbondata.core.constants
+ |.CarbonCommonConstants.MEMBER_DEFAULT_VAL_ARRAY, $valueIntern)) {
+ | $isNull = true;
+ |}
+ """.stripMargin
+
+ val caseCode = getDictionaryColumnIds(e._2)._3 match {
+ case DataType.INT =>
+ s"""
+ |int $value = Integer.parseInt(new String($valueIntern,
+ |org.apache.carbondata.core.constants.CarbonCommonConstants
+ |.DEFAULT_CHARSET_CLASS));
+ """.stripMargin
+ case DataType.SHORT =>
+ s"""
+ |short $value =
+ |Short.parseShort(new String($valueIntern,
+ |org.apache.carbondata.core.constants.CarbonCommonConstants
+ |.DEFAULT_CHARSET_CLASS));
+ """.stripMargin
+ case DataType.DOUBLE =>
+ s"""
+ |double $value =
+ |Double.parseDouble(new String($valueIntern,
+ |org.apache.carbondata.core.constants.CarbonCommonConstants
+ |.DEFAULT_CHARSET_CLASS));
+ """.stripMargin
+ case DataType.LONG =>
+ s"""
+ |long $value =
+ |Long.parseLong(new String($valueIntern,
+ |org.apache.carbondata.core.constants.CarbonCommonConstants
+ |.DEFAULT_CHARSET_CLASS));
+ """.stripMargin
+ case DataType.DECIMAL =>
+ s"""
+ |org.apache.spark.sql.types.Decimal $value =
+ |Decimal.apply(new java.math.BigDecimal(
+ |new String($valueIntern, org.apache.carbondata.core.constants
+ |.CarbonCommonConstants.DEFAULT_CHARSET_CLASS)));
+ """.stripMargin
+ case _ =>
+ s"""
+ | UTF8String $value = UTF8String.fromBytes($valueIntern);
+ """.stripMargin
+ }
+ code +=
+ s"""
+ |$caseCode
+ """.stripMargin
--- End diff --
simplify it in one line
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #653: [CARBONDATA-769] Added codegen suppo...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/653#discussion_r105921624
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/CarbonDictionaryDecoder.scala ---
@@ -225,6 +343,25 @@ case class CarbonDictionaryDecoder(
dicts
}
+ private def getDictionaryWrapper(atiMap: Map[String, AbsoluteTableIdentifier],
+ cache: Cache[DictionaryColumnUniqueIdentifier, Dictionary], storePath: String) = {
+ val dicts: Seq[ForwardDictionaryWrapper] = getDictionaryColumnIds.map { f =>
--- End diff --
suggest to change `f` to a more meaningful variable
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #653: [CARBONDATA-769] Added codegen suppo...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/653#discussion_r106149905
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/CarbonDictionaryDecoder.scala ---
@@ -200,6 +201,123 @@ case class CarbonDictionaryDecoder(
}
}
+ override def doConsume(ctx: CodegenContext, input: Seq[ExprCode], row: ExprCode): String = {
+
+ val storePath = CarbonEnv.get.carbonMetastore.storePath
+ val absoluteTableIdentifiers = relations.map { relation =>
+ val carbonTable = relation.carbonRelation.carbonRelation.metaData.carbonTable
+ (carbonTable.getFactTableName, carbonTable.getAbsoluteTableIdentifier)
+ }.toMap
+
+ if (isRequiredToDecode) {
+ val cacheProvider: CacheProvider = CacheProvider.getInstance
+ val forwardDictionaryCache: Cache[DictionaryColumnUniqueIdentifier, Dictionary] =
+ cacheProvider.createCache(CacheType.FORWARD_DICTIONARY, storePath)
+ val dicts: Seq[ForwardDictionaryWrapper] = getDictionaryWrapper(absoluteTableIdentifiers,
+ forwardDictionaryCache, storePath)
+
+ val exprs = child.output.map(x =>
--- End diff --
ok
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #653: [CARBONDATA-769] Added codegen support to s...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/incubator-carbondata/pull/653
Build Success with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1159/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #653: [CARBONDATA-769] Added codegen support to s...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/incubator-carbondata/pull/653
Build Success with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1161/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #653: [CARBONDATA-769] Added codegen suppo...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/653#discussion_r105921062
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/CarbonDictionaryDecoder.scala ---
@@ -200,6 +201,123 @@ case class CarbonDictionaryDecoder(
}
}
+ override def doConsume(ctx: CodegenContext, input: Seq[ExprCode], row: ExprCode): String = {
+
+ val storePath = CarbonEnv.get.carbonMetastore.storePath
+ val absoluteTableIdentifiers = relations.map { relation =>
+ val carbonTable = relation.carbonRelation.carbonRelation.metaData.carbonTable
+ (carbonTable.getFactTableName, carbonTable.getAbsoluteTableIdentifier)
+ }.toMap
+
+ if (isRequiredToDecode) {
+ val cacheProvider: CacheProvider = CacheProvider.getInstance
+ val forwardDictionaryCache: Cache[DictionaryColumnUniqueIdentifier, Dictionary] =
+ cacheProvider.createCache(CacheType.FORWARD_DICTIONARY, storePath)
+ val dicts: Seq[ForwardDictionaryWrapper] = getDictionaryWrapper(absoluteTableIdentifiers,
+ forwardDictionaryCache, storePath)
+
+ val exprs = child.output.map(x =>
+ ExpressionCanonicalizer.execute(BindReferences.bindReference(x, child.output)))
+ ctx.currentVars = input
+ val resultVars = exprs.zipWithIndex.map { e =>
--- End diff --
change `e` to `case (expr, index)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #653: [CARBONDATA-769] Added codegen suppo...
Posted by ravipesala <gi...@git.apache.org>.
GitHub user ravipesala reopened a pull request:
https://github.com/apache/incubator-carbondata/pull/653
[CARBONDATA-769] Added codegen support to spark2.1 carbon dictionary decoder.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ravipesala/incubator-carbondata measure-filter
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-carbondata/pull/653.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #653
----
commit 91104245c482b93b8bdf90775a083a401544f26e
Author: ravipesala <ra...@gmail.com>
Date: 2017-03-14T07:44:38Z
Added codegen support to spark2.1 carbon decoder.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #653: [CARBONDATA-769] Added codegen suppo...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/653#discussion_r106149917
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/CarbonDictionaryDecoder.scala ---
@@ -225,6 +343,25 @@ case class CarbonDictionaryDecoder(
dicts
}
+ private def getDictionaryWrapper(atiMap: Map[String, AbsoluteTableIdentifier],
+ cache: Cache[DictionaryColumnUniqueIdentifier, Dictionary], storePath: String) = {
+ val dicts: Seq[ForwardDictionaryWrapper] = getDictionaryColumnIds.map { f =>
--- End diff --
ok
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---