You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Loudongfeng <lo...@huawei.com> on 2015/05/26 05:54:35 UTC
Big Lock in Driver.compileInternal
Hi, All
I notice that there is a big lock in org.apache.hadoop.hive.ql.Driver
Following is a piece of code from Apache Hive 1.2.0
private static final Object compileMonitor = new Object();
private int compileInternal(String command) {
int ret;
synchronized (compileMonitor) {
ret = compile(command);
}
...
}
This means HQLs submitted concurrently from clients side will be compiled one by one on Hive Server side.
This will cause problem when compile phase is slow.
My question is ,what does this lock protect for? Is it possible to remove it ?
Best Regards
Nemon
Re: Big Lock in Driver.compileInternal
Posted by Loudongfeng <lo...@huawei.com>.
Glad to know it.
I have tested it yesterday on a 7 nodes cluster, with 100 concurrent clients running for about 8 hours and haven't seen any issue.
The workload is TPC-H style with both DDLs and queries. The 100 clients use different tables from each other and use different sessions.
My Hive version is 0.13.1.And the changing part is the removal of the big lock.
Hope this test helps.
----- -----
Hi. As luck would have it, we are currently looking at this issue :) I have a small patch up at https://issues.apache.org/jira/browse/HIVE-4239; I tested it a bit w/a unit test and some manual cluster testing. Would you be willing to test it on your setup?
On 15/5/25, 20:54, "Loudongfeng" <lo...@huawei.com> wrote:
>Hi, All
>
>I notice that there is a big lock in org.apache.hadoop.hive.ql.Driver
>Following is a piece of code from Apache Hive 1.2.0
>
>private static final Object compileMonitor = new Object();
>
>private int compileInternal(String command) {
> int ret;
> synchronized (compileMonitor) {
> ret = compile(command);
> }
>...
>}
>
>This means HQLs submitted concurrently from clients side will be
>compiled one by one on Hive Server side.
>This will cause problem when compile phase is slow.
>
>My question is ,what does this lock protect for? Is it possible to
>remove it ?
>
>Best Regards
>Nemon
Re: Big Lock in Driver.compileInternal
Posted by Sergey Shelukhin <se...@hortonworks.com>.
Hi. As luck would have it, we are currently looking at this issue :)
I have a small patch up at
https://issues.apache.org/jira/browse/HIVE-4239; I tested it a bit w/a
unit test and some manual cluster testing. Would you be willing to test it
on your setup?
On 15/5/25, 20:54, "Loudongfeng" <lo...@huawei.com> wrote:
>Hi, All
>
>I notice that there is a big lock in org.apache.hadoop.hive.ql.Driver
>Following is a piece of code from Apache Hive 1.2.0
>
>private static final Object compileMonitor = new Object();
>
>private int compileInternal(String command) {
> int ret;
> synchronized (compileMonitor) {
> ret = compile(command);
> }
>...
>}
>
>This means HQLs submitted concurrently from clients side will be compiled
>one by one on Hive Server side.
>This will cause problem when compile phase is slow.
>
>My question is ,what does this lock protect for? Is it possible to remove
>it ?
>
>Best Regards
>Nemon