You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@carbondata.apache.org by QiangCai <gi...@git.apache.org> on 2016/09/22 12:56:42 UTC
[GitHub] incubator-carbondata pull request #190: [CARBONDATA-268]Improve carbonoptimi...
GitHub user QiangCai opened a pull request:
https://github.com/apache/incubator-carbondata/pull/190
[CARBONDATA-268]Improve carbonoptimizer
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/QiangCai/incubator-carbondata improveoptimizer
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-carbondata/pull/190.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #190
----
commit 343a3d20fca9bee15a22063b4be8d73d21b81646
Author: c00318382 <c0...@huaweiobz.com>
Date: 2016-09-22T10:57:00Z
improve carbonoptimizer
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #190: [CARBONDATA-268]Improve carbonoptimi...
Posted by gvramana <gi...@git.apache.org>.
Github user gvramana commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/190#discussion_r80219576
--- Diff: integration/spark/src/main/scala/org/apache/spark/sql/optimizer/CarbonOptimizer.scala ---
@@ -87,20 +130,32 @@ class ResolveCarbonFunctions(relations: Seq[CarbonDecoderRelation])
return plan
}
var decoder = false
+ val mapOfNodes = new java.util.HashMap[LogicalPlan, ExtraNodeInfo]
+ fillNodeInfo(plan, mapOfNodes)
val aliasMap = CarbonAliasDecoderRelation()
// collect alias information before hand.
collectInformationOnAttributes(plan, aliasMap)
- val transFormedPlan =
- plan transformDown {
- case cd: CarbonDictionaryTempDecoder if cd.isOuter =>
- decoder = true
- cd
+
+ def hasCarbonRelation(currentPlan: LogicalPlan): Boolean = {
+ val extraNodeInfo = mapOfNodes.get(currentPlan)
--- End diff --
Change the name of mapofNodes to mapOfNonCarbonPlanNodes
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #190: [CARBONDATA-268]Improve carbonoptimi...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-carbondata/pull/190
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #190: [CARBONDATA-268]Improve carbonoptimi...
Posted by gvramana <gi...@git.apache.org>.
Github user gvramana commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/190#discussion_r80220071
--- Diff: integration/spark/src/main/scala/org/apache/spark/sql/optimizer/CarbonOptimizer.scala ---
@@ -68,9 +70,50 @@ object CarbonOptimizer {
*/
class ResolveCarbonFunctions(relations: Seq[CarbonDecoderRelation])
extends Rule[LogicalPlan] with PredicateHelper {
-
+ val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
def apply(plan: LogicalPlan): LogicalPlan = {
- transformCarbonPlan(plan, relations)
+ if (relations.nonEmpty && !isOptimized(plan)) {
+ LOGGER.info("Starting to optimize plan")
+ val startTime = System.currentTimeMillis
+ val result = transformCarbonPlan(plan, relations)
+ LOGGER.info("Time taken to optimize plan: " + ( System.currentTimeMillis - startTime))
--- End diff --
add this to STATISTICS logger type
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #190: [CARBONDATA-268]Improve carbonoptimi...
Posted by gvramana <gi...@git.apache.org>.
Github user gvramana commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/190#discussion_r80220168
--- Diff: integration/spark/src/main/scala/org/apache/spark/sql/optimizer/CarbonOptimizer.scala ---
@@ -397,20 +452,38 @@ class ResolveCarbonFunctions(relations: Seq[CarbonDecoderRelation])
case others => others
}
+ }
+
+ val tStartTime = System.currentTimeMillis
--- End diff --
this variable not used
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #190: [CARBONDATA-268]Improve carbonoptimi...
Posted by gvramana <gi...@git.apache.org>.
Github user gvramana commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/190#discussion_r80219989
--- Diff: integration/spark/src/main/scala/org/apache/spark/sql/optimizer/CarbonOptimizer.scala ---
@@ -68,9 +70,50 @@ object CarbonOptimizer {
*/
class ResolveCarbonFunctions(relations: Seq[CarbonDecoderRelation])
extends Rule[LogicalPlan] with PredicateHelper {
-
+ val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
def apply(plan: LogicalPlan): LogicalPlan = {
- transformCarbonPlan(plan, relations)
+ if (relations.nonEmpty && !isOptimized(plan)) {
+ LOGGER.info("Starting to optimize plan")
+ val startTime = System.currentTimeMillis
+ val result = transformCarbonPlan(plan, relations)
+ LOGGER.info("Time taken to optimize plan: " + ( System.currentTimeMillis - startTime))
--- End diff --
change the statement to "Time taken for Carbon Optimizer to optimize"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---