You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by XIAOBING ZHOU <xz...@gmail.com> on 2015/02/21 02:05:17 UTC
Review Request 31249: HIVE-9741: Refactor MetaStoreDirectSql by using
getProductName instead of querying DB to determine DbType
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31249/
-----------------------------------------------------------
Review request for hive.
Repository: hive-git
Description
-------
MetaStoreDirectSql constructor is querying DB to determine dbType. which leads to too many DB queries to make metastore slow or hanging if MetaStoreDirectSql constructor is frequently called. This is to propose getProductName to get dbType info.
Diffs
-----
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 574141c
Diff: https://reviews.apache.org/r/31249/diff/
Testing
-------
Thanks,
XIAOBING ZHOU
Re: Review Request 31249: Refactor MetaStoreDirectSql constructor by
removing DB queries out of critical section
Posted by Sergey Shelukhin <se...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31249/#review73950
-----------------------------------------------------------
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
<https://reviews.apache.org/r/31249/#comment120366>
this logging statement is invalid now it appears
- Sergey Shelukhin
On Feb. 24, 2015, 10:34 p.m., XIAOBING ZHOU wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31249/
> -----------------------------------------------------------
>
> (Updated Feb. 24, 2015, 10:34 p.m.)
>
>
> Review request for hive.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> MetaStoreDirectSql constructor is querying DB to determine dbType, which leads to too many DB queries to make megastore slow as ObjectStore.setConf might be called frequently. Moreover, ObjectStore.setConf begins/ends with lock acquire/release, if the underlying DB hangs somehow, lock is never released and all hereafter incoming requests are blocked.
>
> Two points:
> 1. Using getProductName based JDBC driver to get dbType info.
> 2. Since metastore auto-creaton is disabled by default, it'd better bypass ensureDbInit() and runTestQuery() in order to avoid DB queries within critical section of setConf.
>
> Here’s stack trace:
> MetaStoreDirectSql.determineDbType(...)
> MetaStoreDirectSql.MetaStoreDirectSql(...)
> ObjectStore.initialize(...)
> ObjectStore.setConf(…)
>
>
> Diffs
> -----
>
> metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 574141c
>
> Diff: https://reviews.apache.org/r/31249/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> XIAOBING ZHOU
>
>
Re: Review Request 31249: Refactor MetaStoreDirectSql constructor by
removing DB queries out of critical section
Posted by XIAOBING ZHOU <xz...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31249/
-----------------------------------------------------------
(Updated Feb. 26, 2015, 8:53 p.m.)
Review request for hive.
Changes
-------
added doDbSpecificInitializationsBeforeQuery in runTestQuery
Repository: hive-git
Description
-------
MetaStoreDirectSql constructor is querying DB to determine dbType, which leads to too many DB queries to make megastore slow as ObjectStore.setConf might be called frequently. Moreover, ObjectStore.setConf begins/ends with lock acquire/release, if the underlying DB hangs somehow, lock is never released and all hereafter incoming requests are blocked.
Two points:
1. Using getProductName based JDBC driver to get dbType info.
2. Since metastore auto-creaton is disabled by default, it'd better bypass ensureDbInit() and runTestQuery() in order to avoid DB queries within critical section of setConf.
Here’s stack trace:
MetaStoreDirectSql.determineDbType(...)
MetaStoreDirectSql.MetaStoreDirectSql(...)
ObjectStore.initialize(...)
ObjectStore.setConf(…)
https://issues.apache.org/jira/browse/HIVE-9741
Diffs (updated)
-----
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 574141c
Diff: https://reviews.apache.org/r/31249/diff/
Testing
-------
Thanks,
XIAOBING ZHOU
Re: Review Request 31249: Refactor MetaStoreDirectSql constructor by
removing DB queries out of critical section
Posted by XIAOBING ZHOU <xz...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31249/
-----------------------------------------------------------
(Updated Feb. 25, 2015, 11:58 p.m.)
Review request for hive.
Changes
-------
remved setDbANSIMode
Repository: hive-git
Description
-------
MetaStoreDirectSql constructor is querying DB to determine dbType, which leads to too many DB queries to make megastore slow as ObjectStore.setConf might be called frequently. Moreover, ObjectStore.setConf begins/ends with lock acquire/release, if the underlying DB hangs somehow, lock is never released and all hereafter incoming requests are blocked.
Two points:
1. Using getProductName based JDBC driver to get dbType info.
2. Since metastore auto-creaton is disabled by default, it'd better bypass ensureDbInit() and runTestQuery() in order to avoid DB queries within critical section of setConf.
Here’s stack trace:
MetaStoreDirectSql.determineDbType(...)
MetaStoreDirectSql.MetaStoreDirectSql(...)
ObjectStore.initialize(...)
ObjectStore.setConf(…)
https://issues.apache.org/jira/browse/HIVE-9741
Diffs (updated)
-----
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 574141c
Diff: https://reviews.apache.org/r/31249/diff/
Testing
-------
Thanks,
XIAOBING ZHOU
Re: Review Request 31249: Refactor MetaStoreDirectSql constructor by
removing DB queries out of critical section
Posted by XIAOBING ZHOU <xz...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31249/
-----------------------------------------------------------
(Updated Feb. 25, 2015, 9:37 p.m.)
Review request for hive.
Changes
-------
1. moved setDbANSIMode call
2. removed logging in comment
Repository: hive-git
Description
-------
MetaStoreDirectSql constructor is querying DB to determine dbType, which leads to too many DB queries to make megastore slow as ObjectStore.setConf might be called frequently. Moreover, ObjectStore.setConf begins/ends with lock acquire/release, if the underlying DB hangs somehow, lock is never released and all hereafter incoming requests are blocked.
Two points:
1. Using getProductName based JDBC driver to get dbType info.
2. Since metastore auto-creaton is disabled by default, it'd better bypass ensureDbInit() and runTestQuery() in order to avoid DB queries within critical section of setConf.
Here’s stack trace:
MetaStoreDirectSql.determineDbType(...)
MetaStoreDirectSql.MetaStoreDirectSql(...)
ObjectStore.initialize(...)
ObjectStore.setConf(…)
https://issues.apache.org/jira/browse/HIVE-9741
Diffs (updated)
-----
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 574141c
Diff: https://reviews.apache.org/r/31249/diff/
Testing
-------
Thanks,
XIAOBING ZHOU
Re: Review Request 31249: Refactor MetaStoreDirectSql constructor by
removing DB queries out of critical section
Posted by XIAOBING ZHOU <xz...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31249/
-----------------------------------------------------------
(Updated Feb. 25, 2015, 9:24 p.m.)
Review request for hive.
Repository: hive-git
Description (updated)
-------
MetaStoreDirectSql constructor is querying DB to determine dbType, which leads to too many DB queries to make megastore slow as ObjectStore.setConf might be called frequently. Moreover, ObjectStore.setConf begins/ends with lock acquire/release, if the underlying DB hangs somehow, lock is never released and all hereafter incoming requests are blocked.
Two points:
1. Using getProductName based JDBC driver to get dbType info.
2. Since metastore auto-creaton is disabled by default, it'd better bypass ensureDbInit() and runTestQuery() in order to avoid DB queries within critical section of setConf.
Here’s stack trace:
MetaStoreDirectSql.determineDbType(...)
MetaStoreDirectSql.MetaStoreDirectSql(...)
ObjectStore.initialize(...)
ObjectStore.setConf(…)
https://issues.apache.org/jira/browse/HIVE-9741
Diffs
-----
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 574141c
Diff: https://reviews.apache.org/r/31249/diff/
Testing
-------
Thanks,
XIAOBING ZHOU
Re: Review Request 31249: Refactor MetaStoreDirectSql constructor by
removing DB queries out of critical section
Posted by XIAOBING ZHOU <xz...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31249/
-----------------------------------------------------------
(Updated Feb. 24, 2015, 10:34 p.m.)
Review request for hive.
Changes
-------
removed empty spaces.
Repository: hive-git
Description
-------
MetaStoreDirectSql constructor is querying DB to determine dbType, which leads to too many DB queries to make megastore slow as ObjectStore.setConf might be called frequently. Moreover, ObjectStore.setConf begins/ends with lock acquire/release, if the underlying DB hangs somehow, lock is never released and all hereafter incoming requests are blocked.
Two points:
1. Using getProductName based JDBC driver to get dbType info.
2. Since metastore auto-creaton is disabled by default, it'd better bypass ensureDbInit() and runTestQuery() in order to avoid DB queries within critical section of setConf.
Here’s stack trace:
MetaStoreDirectSql.determineDbType(...)
MetaStoreDirectSql.MetaStoreDirectSql(...)
ObjectStore.initialize(...)
ObjectStore.setConf(…)
Diffs (updated)
-----
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 574141c
Diff: https://reviews.apache.org/r/31249/diff/
Testing
-------
Thanks,
XIAOBING ZHOU
Re: Review Request 31249: Refactor MetaStoreDirectSql constructor by
removing DB queries out of critical section
Posted by XIAOBING ZHOU <xz...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31249/
-----------------------------------------------------------
(Updated Feb. 24, 2015, 10:24 p.m.)
Review request for hive.
Changes
-------
moved setDbANSIMode call to MetaStoreDirectSql
Repository: hive-git
Description
-------
MetaStoreDirectSql constructor is querying DB to determine dbType, which leads to too many DB queries to make megastore slow as ObjectStore.setConf might be called frequently. Moreover, ObjectStore.setConf begins/ends with lock acquire/release, if the underlying DB hangs somehow, lock is never released and all hereafter incoming requests are blocked.
Two points:
1. Using getProductName based JDBC driver to get dbType info.
2. Since metastore auto-creaton is disabled by default, it'd better bypass ensureDbInit() and runTestQuery() in order to avoid DB queries within critical section of setConf.
Here’s stack trace:
MetaStoreDirectSql.determineDbType(...)
MetaStoreDirectSql.MetaStoreDirectSql(...)
ObjectStore.initialize(...)
ObjectStore.setConf(…)
Diffs (updated)
-----
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 574141c
Diff: https://reviews.apache.org/r/31249/diff/
Testing
-------
Thanks,
XIAOBING ZHOU
Re: Review Request 31249: Refactor MetaStoreDirectSql constructor by
removing DB queries out of critical section
Posted by XIAOBING ZHOU <xz...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31249/
-----------------------------------------------------------
(Updated Feb. 24, 2015, 12:49 a.m.)
Review request for hive.
Summary (updated)
-----------------
Refactor MetaStoreDirectSql constructor by removing DB queries out of critical section
Repository: hive-git
Description
-------
MetaStoreDirectSql constructor is querying DB to determine dbType, which leads to too many DB queries to make megastore slow as ObjectStore.setConf might be called frequently. Moreover, ObjectStore.setConf begins/ends with lock acquire/release, if the underlying DB hangs somehow, lock is never released and all hereafter incoming requests are blocked.
Two points:
1. Using getProductName based JDBC driver to get dbType info.
2. Since metastore auto-creaton is disabled by default, it'd better bypass ensureDbInit() and runTestQuery() in order to avoid DB queries within critical section of setConf.
Here’s stack trace:
MetaStoreDirectSql.determineDbType(...)
MetaStoreDirectSql.MetaStoreDirectSql(...)
ObjectStore.initialize(...)
ObjectStore.setConf(…)
Diffs
-----
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 574141c
metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java fcaffc7
Diff: https://reviews.apache.org/r/31249/diff/
Testing
-------
Thanks,
XIAOBING ZHOU
Re: Review Request 31249: Refactor MetaStoreDirectSql constructor
Posted by XIAOBING ZHOU <xz...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31249/
-----------------------------------------------------------
(Updated Feb. 23, 2015, 10:08 p.m.)
Review request for hive.
Changes
-------
Two points:
1. Using getProductName based JDBC driver to get dbType info.
2. Since metastore auto-creaton is disabled by default, it'd better bypass ensureDbInit() and runTestQuery() in order to avoid DB queries within critical section of setConf.
Summary (updated)
-----------------
Refactor MetaStoreDirectSql constructor
Repository: hive-git
Description (updated)
-------
MetaStoreDirectSql constructor is querying DB to determine dbType, which leads to too many DB queries to make megastore slow as ObjectStore.setConf might be called frequently. Moreover, ObjectStore.setConf begins/ends with lock acquire/release, if the underlying DB hangs somehow, lock is never released and all hereafter incoming requests are blocked.
Two points:
1. Using getProductName based JDBC driver to get dbType info.
2. Since metastore auto-creaton is disabled by default, it'd better bypass ensureDbInit() and runTestQuery() in order to avoid DB queries within critical section of setConf.
Here’s stack trace:
MetaStoreDirectSql.determineDbType(...)
MetaStoreDirectSql.MetaStoreDirectSql(...)
ObjectStore.initialize(...)
ObjectStore.setConf(…)
Diffs (updated)
-----
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 574141c
metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java fcaffc7
Diff: https://reviews.apache.org/r/31249/diff/
Testing
-------
Thanks,
XIAOBING ZHOU