You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by xuchuanyin <gi...@git.apache.org> on 2017/07/27 07:09:44 UTC
[GitHub] carbondata pull request #1199: [CARBONDATA-1335] Remove duplicated time-cons...
GitHub user xuchuanyin opened a pull request:
https://github.com/apache/carbondata/pull/1199
[CARBONDATA-1335] Remove duplicated time-consuming method call
# Scenario
Currently we did a concurrent 14 queries on Carbondata. The queries are the same, but on different tables. We have noticed the following scene:
+ A single query took about 5s;
+ In concurrent scenario, each query took about 15s;
By adding checkpoint in the log, we found that there was great latency in starting query jobs in spark.
# Analysts
When we fire a query, Carbondata firstly do some job in the client side, including parse/analyze plans and prepare filtered blocks and inputSplits. Then Carbondata start to submit query job to spark.
We found in the first step, Carbondata took about 7s in current scenario, but it only took about <1s in single scenario.
By studying the related code, we found the most time consuming method call was `CarbonSessionCatalog.lookupRelation`. In side this method, it called `super.lookupRelation` twice, which consumed about 3s each time.
# Solution
Carbondata only needs to call the `super.lookupRelation` only once, we need to remove the useless duplicated method call.
I've tested in my environment and it works well. In concurrent scenario, each query takes about 12s (3s saved for the improvement).
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/xuchuanyin/carbondata remove_duplicated_lookup
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/1199.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1199
----
commit d05d1fadbe0f773a00ff1d0e96ff9fe90b7b7f06
Author: xuchuanyin <xu...@hust.edu.cn>
Date: 2017-07-27T07:07:25Z
Remove duplicated time-consuming method call
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #1199: [CARBONDATA-1335] Remove duplicated time-consuming m...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1199
Build Success with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/617/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #1199: [CARBONDATA-1335] Remove duplicated time-consuming m...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1199
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/3212/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #1199: [CARBONDATA-1335] Remove duplicated time-consuming m...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on the issue:
https://github.com/apache/carbondata/pull/1199
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #1199: [CARBONDATA-1335] Remove duplicated time-cons...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1199#discussion_r129767934
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonSessionState.scala ---
@@ -106,7 +109,6 @@ class CarbonSessionCatalog(
}
case _ =>
}
- super.lookupRelation(name, alias)
--- End diff --
this PR mainly focus on removing this useless method call
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #1199: [CARBONDATA-1335] Remove duplicated time-consuming m...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/carbondata/pull/1199
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #1199: [CARBONDATA-1335] Remove duplicated time-cons...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/carbondata/pull/1199
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---