You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by jinfengni <gi...@git.apache.org> on 2015/12/01 19:57:22 UTC
[GitHub] drill pull request: Drill 4127: Reduce Hive metastore client API c...
GitHub user jinfengni opened a pull request:
https://github.com/apache/drill/pull/286
Drill 4127: Reduce Hive metastore client API call in HiveSchema
Also, it has commit for DRILL-4126: Add cache to HiveSchema in order to reduce long planning time or execution time caused by slow Hive meta store.
Both DRILL-4127 and DRILL-4126 address the long delay caused by slow hive meta store.
Passed unit, pre-commit regression, and additional impersonation test, before rebasing onto latest master.
Will re-run the above tests.
@vkorukanti , could you please review the two patches? Thanks.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jinfengni/incubator-drill DRILL-4127
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/drill/pull/286.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #286
----
commit 19a5a4d1c9c23eedcb94c988bd2229680575a118
Author: Jinfeng Ni <jn...@apache.org>
Date: 2015-11-19T04:18:51Z
DRILL-4127: Reduce Hive metastore client API call in HiveSchema.
1) Use lazy loading of tableNames in HiveSchema, in stead of pre-loading all table names under each HiveSchema.
2) Do not call get_all_databases for subSchema to check existence if the name comes from getSubSchemaNames() directly.
commit 9570319c227649144d3a14f8d5774fbe4a282bc4
Author: Jinfeng Ni <jn...@apache.org>
Date: 2015-11-30T04:15:07Z
DRILL-4126: Add cache to HiveSchema in order to reduce long planning time or execution time caused by slow Hive meta store.
1) HiveSchema caching will help in case impersonation is enabled.
2) Use flat level cache for tables in DrillHiveMetaStoreClient.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] drill pull request: Drill 4127: Reduce Hive metastore client API c...
Posted by jinfengni <gi...@git.apache.org>.
Github user jinfengni commented on the pull request:
https://github.com/apache/drill/pull/286#issuecomment-161210859
@vkorukanti , I revised the patch based on our discussions.
The cache is enabled when impersonation is turned on. I also put another commit which uses flat level cache for tables. I run the queries which originally hit the long planning time/execution time, and seems things are what I expected.
Please take a look at the PR. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] drill pull request: Drill 4127: Reduce Hive metastore client API c...
Posted by jinfengni <gi...@git.apache.org>.
Github user jinfengni commented on the pull request:
https://github.com/apache/drill/pull/286#issuecomment-161335544
@vkorukanti , the patches pass the unit test, pre-commit test, and impersonation test.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] drill pull request: Drill 4127: Reduce Hive metastore client API c...
Posted by jinfengni <gi...@git.apache.org>.
Github user jinfengni closed the pull request at:
https://github.com/apache/drill/pull/286
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] drill pull request: Drill 4127: Reduce Hive metastore client API c...
Posted by vkorukanti <gi...@git.apache.org>.
Github user vkorukanti commented on the pull request:
https://github.com/apache/drill/pull/286#issuecomment-162705796
Have some minor comments on the DRILL-4127. LGTM, +1.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] drill pull request: Drill 4127: Reduce Hive metastore client API c...
Posted by jinfengni <gi...@git.apache.org>.
Github user jinfengni commented on the pull request:
https://github.com/apache/drill/pull/286#issuecomment-162665850
@vkorukanti , could you please take a look at the revised patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] drill pull request: Drill 4127: Reduce Hive metastore client API c...
Posted by jinfengni <gi...@git.apache.org>.
Github user jinfengni commented on the pull request:
https://github.com/apache/drill/pull/286#issuecomment-162982945
Address venki's comments in the revised patches.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---