You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by XIAOBING ZHOU <xz...@gmail.com> on 2015/02/24 01:49:15 UTC
Re: Review Request 31249: Refactor MetaStoreDirectSql constructor by
removing DB queries out of critical section
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31249/
-----------------------------------------------------------
(Updated Feb. 24, 2015, 12:49 a.m.)
Review request for hive.
Summary (updated)
-----------------
Refactor MetaStoreDirectSql constructor by removing DB queries out of critical section
Repository: hive-git
Description
-------
MetaStoreDirectSql constructor is querying DB to determine dbType, which leads to too many DB queries to make megastore slow as ObjectStore.setConf might be called frequently. Moreover, ObjectStore.setConf begins/ends with lock acquire/release, if the underlying DB hangs somehow, lock is never released and all hereafter incoming requests are blocked.
Two points:
1. Using getProductName based JDBC driver to get dbType info.
2. Since metastore auto-creaton is disabled by default, it'd better bypass ensureDbInit() and runTestQuery() in order to avoid DB queries within critical section of setConf.
Here’s stack trace:
MetaStoreDirectSql.determineDbType(...)
MetaStoreDirectSql.MetaStoreDirectSql(...)
ObjectStore.initialize(...)
ObjectStore.setConf(…)
Diffs
-----
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 574141c
metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java fcaffc7
Diff: https://reviews.apache.org/r/31249/diff/
Testing
-------
Thanks,
XIAOBING ZHOU
Re: Review Request 31249: Refactor MetaStoreDirectSql constructor by
removing DB queries out of critical section
Posted by Sergey Shelukhin <se...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31249/#review73950
-----------------------------------------------------------
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
<https://reviews.apache.org/r/31249/#comment120366>
this logging statement is invalid now it appears
- Sergey Shelukhin
On Feb. 24, 2015, 10:34 p.m., XIAOBING ZHOU wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31249/
> -----------------------------------------------------------
>
> (Updated Feb. 24, 2015, 10:34 p.m.)
>
>
> Review request for hive.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> MetaStoreDirectSql constructor is querying DB to determine dbType, which leads to too many DB queries to make megastore slow as ObjectStore.setConf might be called frequently. Moreover, ObjectStore.setConf begins/ends with lock acquire/release, if the underlying DB hangs somehow, lock is never released and all hereafter incoming requests are blocked.
>
> Two points:
> 1. Using getProductName based JDBC driver to get dbType info.
> 2. Since metastore auto-creaton is disabled by default, it'd better bypass ensureDbInit() and runTestQuery() in order to avoid DB queries within critical section of setConf.
>
> Here’s stack trace:
> MetaStoreDirectSql.determineDbType(...)
> MetaStoreDirectSql.MetaStoreDirectSql(...)
> ObjectStore.initialize(...)
> ObjectStore.setConf(…)
>
>
> Diffs
> -----
>
> metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 574141c
>
> Diff: https://reviews.apache.org/r/31249/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> XIAOBING ZHOU
>
>
Re: Review Request 31249: Refactor MetaStoreDirectSql constructor by
removing DB queries out of critical section
Posted by XIAOBING ZHOU <xz...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31249/
-----------------------------------------------------------
(Updated Feb. 26, 2015, 8:53 p.m.)
Review request for hive.
Changes
-------
added doDbSpecificInitializationsBeforeQuery in runTestQuery
Repository: hive-git
Description
-------
MetaStoreDirectSql constructor is querying DB to determine dbType, which leads to too many DB queries to make megastore slow as ObjectStore.setConf might be called frequently. Moreover, ObjectStore.setConf begins/ends with lock acquire/release, if the underlying DB hangs somehow, lock is never released and all hereafter incoming requests are blocked.
Two points:
1. Using getProductName based JDBC driver to get dbType info.
2. Since metastore auto-creaton is disabled by default, it'd better bypass ensureDbInit() and runTestQuery() in order to avoid DB queries within critical section of setConf.
Here’s stack trace:
MetaStoreDirectSql.determineDbType(...)
MetaStoreDirectSql.MetaStoreDirectSql(...)
ObjectStore.initialize(...)
ObjectStore.setConf(…)
https://issues.apache.org/jira/browse/HIVE-9741
Diffs (updated)
-----
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 574141c
Diff: https://reviews.apache.org/r/31249/diff/
Testing
-------
Thanks,
XIAOBING ZHOU
Re: Review Request 31249: Refactor MetaStoreDirectSql constructor by
removing DB queries out of critical section
Posted by XIAOBING ZHOU <xz...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31249/
-----------------------------------------------------------
(Updated Feb. 25, 2015, 11:58 p.m.)
Review request for hive.
Changes
-------
remved setDbANSIMode
Repository: hive-git
Description
-------
MetaStoreDirectSql constructor is querying DB to determine dbType, which leads to too many DB queries to make megastore slow as ObjectStore.setConf might be called frequently. Moreover, ObjectStore.setConf begins/ends with lock acquire/release, if the underlying DB hangs somehow, lock is never released and all hereafter incoming requests are blocked.
Two points:
1. Using getProductName based JDBC driver to get dbType info.
2. Since metastore auto-creaton is disabled by default, it'd better bypass ensureDbInit() and runTestQuery() in order to avoid DB queries within critical section of setConf.
Here’s stack trace:
MetaStoreDirectSql.determineDbType(...)
MetaStoreDirectSql.MetaStoreDirectSql(...)
ObjectStore.initialize(...)
ObjectStore.setConf(…)
https://issues.apache.org/jira/browse/HIVE-9741
Diffs (updated)
-----
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 574141c
Diff: https://reviews.apache.org/r/31249/diff/
Testing
-------
Thanks,
XIAOBING ZHOU
Re: Review Request 31249: Refactor MetaStoreDirectSql constructor by
removing DB queries out of critical section
Posted by XIAOBING ZHOU <xz...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31249/
-----------------------------------------------------------
(Updated Feb. 25, 2015, 9:37 p.m.)
Review request for hive.
Changes
-------
1. moved setDbANSIMode call
2. removed logging in comment
Repository: hive-git
Description
-------
MetaStoreDirectSql constructor is querying DB to determine dbType, which leads to too many DB queries to make megastore slow as ObjectStore.setConf might be called frequently. Moreover, ObjectStore.setConf begins/ends with lock acquire/release, if the underlying DB hangs somehow, lock is never released and all hereafter incoming requests are blocked.
Two points:
1. Using getProductName based JDBC driver to get dbType info.
2. Since metastore auto-creaton is disabled by default, it'd better bypass ensureDbInit() and runTestQuery() in order to avoid DB queries within critical section of setConf.
Here’s stack trace:
MetaStoreDirectSql.determineDbType(...)
MetaStoreDirectSql.MetaStoreDirectSql(...)
ObjectStore.initialize(...)
ObjectStore.setConf(…)
https://issues.apache.org/jira/browse/HIVE-9741
Diffs (updated)
-----
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 574141c
Diff: https://reviews.apache.org/r/31249/diff/
Testing
-------
Thanks,
XIAOBING ZHOU
Re: Review Request 31249: Refactor MetaStoreDirectSql constructor by
removing DB queries out of critical section
Posted by XIAOBING ZHOU <xz...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31249/
-----------------------------------------------------------
(Updated Feb. 25, 2015, 9:24 p.m.)
Review request for hive.
Repository: hive-git
Description (updated)
-------
MetaStoreDirectSql constructor is querying DB to determine dbType, which leads to too many DB queries to make megastore slow as ObjectStore.setConf might be called frequently. Moreover, ObjectStore.setConf begins/ends with lock acquire/release, if the underlying DB hangs somehow, lock is never released and all hereafter incoming requests are blocked.
Two points:
1. Using getProductName based JDBC driver to get dbType info.
2. Since metastore auto-creaton is disabled by default, it'd better bypass ensureDbInit() and runTestQuery() in order to avoid DB queries within critical section of setConf.
Here’s stack trace:
MetaStoreDirectSql.determineDbType(...)
MetaStoreDirectSql.MetaStoreDirectSql(...)
ObjectStore.initialize(...)
ObjectStore.setConf(…)
https://issues.apache.org/jira/browse/HIVE-9741
Diffs
-----
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 574141c
Diff: https://reviews.apache.org/r/31249/diff/
Testing
-------
Thanks,
XIAOBING ZHOU
Re: Review Request 31249: Refactor MetaStoreDirectSql constructor by
removing DB queries out of critical section
Posted by XIAOBING ZHOU <xz...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31249/
-----------------------------------------------------------
(Updated Feb. 24, 2015, 10:34 p.m.)
Review request for hive.
Changes
-------
removed empty spaces.
Repository: hive-git
Description
-------
MetaStoreDirectSql constructor is querying DB to determine dbType, which leads to too many DB queries to make megastore slow as ObjectStore.setConf might be called frequently. Moreover, ObjectStore.setConf begins/ends with lock acquire/release, if the underlying DB hangs somehow, lock is never released and all hereafter incoming requests are blocked.
Two points:
1. Using getProductName based JDBC driver to get dbType info.
2. Since metastore auto-creaton is disabled by default, it'd better bypass ensureDbInit() and runTestQuery() in order to avoid DB queries within critical section of setConf.
Here’s stack trace:
MetaStoreDirectSql.determineDbType(...)
MetaStoreDirectSql.MetaStoreDirectSql(...)
ObjectStore.initialize(...)
ObjectStore.setConf(…)
Diffs (updated)
-----
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 574141c
Diff: https://reviews.apache.org/r/31249/diff/
Testing
-------
Thanks,
XIAOBING ZHOU
Re: Review Request 31249: Refactor MetaStoreDirectSql constructor by
removing DB queries out of critical section
Posted by XIAOBING ZHOU <xz...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31249/
-----------------------------------------------------------
(Updated Feb. 24, 2015, 10:24 p.m.)
Review request for hive.
Changes
-------
moved setDbANSIMode call to MetaStoreDirectSql
Repository: hive-git
Description
-------
MetaStoreDirectSql constructor is querying DB to determine dbType, which leads to too many DB queries to make megastore slow as ObjectStore.setConf might be called frequently. Moreover, ObjectStore.setConf begins/ends with lock acquire/release, if the underlying DB hangs somehow, lock is never released and all hereafter incoming requests are blocked.
Two points:
1. Using getProductName based JDBC driver to get dbType info.
2. Since metastore auto-creaton is disabled by default, it'd better bypass ensureDbInit() and runTestQuery() in order to avoid DB queries within critical section of setConf.
Here’s stack trace:
MetaStoreDirectSql.determineDbType(...)
MetaStoreDirectSql.MetaStoreDirectSql(...)
ObjectStore.initialize(...)
ObjectStore.setConf(…)
Diffs (updated)
-----
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 574141c
Diff: https://reviews.apache.org/r/31249/diff/
Testing
-------
Thanks,
XIAOBING ZHOU