You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Sahil Takiar <ta...@gmail.com> on 2017/04/05 16:11:30 UTC
Re: Review Request 58086: HIVE-16334 Query lock contains the query
string, which can cause OOM on ZooKeeper
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58086/#review171132
-----------------------------------------------------------
Minor comments, overall LGTM
ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestHiveLockObject.java
Lines 57 (patched)
<https://reviews.apache.org/r/58086/#comment244020>
Would be nice to have a test case where the query is exactly the same as the truncation limit.
- Sahil Takiar
On March 30, 2017, 6:01 p.m., Peter Vary wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58086/
> -----------------------------------------------------------
>
> (Updated March 30, 2017, 6:01 p.m.)
>
>
> Review request for hive, Aihua Xu, Chaoyu Tang, Marta Kuczora, Miklos Csanady, and Barna Zsombor Klara.
>
>
> Bugs: HIVE-16334
> https://issues.apache.org/jira/browse/HIVE-16334
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> The patch contains the following changes:
> - Added a new config variable which defines the maximum length of the query string in the lock object
> - When constructing the HiveLockObject gets the configuration as a new parameter
> - When constructing the HiveLockObject truncates the length of the query string if it is necessary
> - Updates every occurance of the HiveLockObject creation
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e12fea1
> ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DummyTxnManager.java 53ee9c8
> ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockObject.java fff03df
> ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveTxnManagerImpl.java a371a5a
> ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDummyTxnManager.java de3b8ad
> ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestEmbeddedLockManager.java 0afbc1c
> ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestHiveLockObject.java 19cb129
> ql/src/test/org/apache/hadoop/hive/ql/lockmgr/zookeeper/TestZookeeperLockManager.java 3f9926e
>
>
> Diff: https://reviews.apache.org/r/58086/diff/1/
>
>
> Testing
> -------
>
> Added new unit test to test the truncation
> Updated the existing unit tests
>
>
> Thanks,
>
> Peter Vary
>
>
Re: Review Request 58086: HIVE-16334 Query lock contains the query
string, which can cause OOM on ZooKeeper
Posted by Peter Vary <pv...@cloudera.com>.
> On April 5, 2017, 4:11 p.m., Sahil Takiar wrote:
> > Minor comments, overall LGTM
Thanks for the reviews Sahil and Vihang!
> On April 5, 2017, 4:11 p.m., Sahil Takiar wrote:
> > ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestHiveLockObject.java
> > Lines 57 (patched)
> > <https://reviews.apache.org/r/58086/diff/1/?file=1681115#file1681115line57>
> >
> > Would be nice to have a test case where the query is exactly the same as the truncation limit.
>
> Vihang Karajgaonkar wrote:
> Adding to Sahil's suggestion above. Can we add a test case to make sure that query string is not truncated when config is -1 to make sure we catch regressions in the future
Added a test to test the default value, and when the query is exactly the same as the truncation limit, thanks for the suggestion Sahil and Vihang!
- Peter
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58086/#review171132
-----------------------------------------------------------
On April 5, 2017, 5:21 p.m., Peter Vary wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58086/
> -----------------------------------------------------------
>
> (Updated April 5, 2017, 5:21 p.m.)
>
>
> Review request for hive, Aihua Xu, Chaoyu Tang, Marta Kuczora, Miklos Csanady, and Barna Zsombor Klara.
>
>
> Bugs: HIVE-16334
> https://issues.apache.org/jira/browse/HIVE-16334
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> The patch contains the following changes:
> - Added a new config variable which defines the maximum length of the query string in the lock object
> - When constructing the HiveLockObject gets the configuration as a new parameter
> - When constructing the HiveLockObject truncates the length of the query string if it is necessary
> - Updates every occurance of the HiveLockObject creation
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 6de6ed6
> ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DummyTxnManager.java 53ee9c8
> ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockObject.java fff03df
> ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveTxnManagerImpl.java a371a5a
> ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDummyTxnManager.java de3b8ad
> ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestEmbeddedLockManager.java 0afbc1c
> ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestHiveLockObject.java 19cb129
> ql/src/test/org/apache/hadoop/hive/ql/lockmgr/zookeeper/TestZookeeperLockManager.java 3f9926e
>
>
> Diff: https://reviews.apache.org/r/58086/diff/2/
>
>
> Testing
> -------
>
> Added new unit test to test the truncation
> Updated the existing unit tests
>
>
> Thanks,
>
> Peter Vary
>
>
Re: Review Request 58086: HIVE-16334 Query lock contains the query
string, which can cause OOM on ZooKeeper
Posted by Vihang Karajgaonkar <vi...@cloudera.com>.
> On April 5, 2017, 4:11 p.m., Sahil Takiar wrote:
> > ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestHiveLockObject.java
> > Lines 57 (patched)
> > <https://reviews.apache.org/r/58086/diff/1/?file=1681115#file1681115line57>
> >
> > Would be nice to have a test case where the query is exactly the same as the truncation limit.
Adding to Sahil's suggestion above. Can we add a test case to make sure that query string is not truncated when config is -1 to make sure we catch regressions in the future
- Vihang
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58086/#review171132
-----------------------------------------------------------
On March 30, 2017, 6:01 p.m., Peter Vary wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58086/
> -----------------------------------------------------------
>
> (Updated March 30, 2017, 6:01 p.m.)
>
>
> Review request for hive, Aihua Xu, Chaoyu Tang, Marta Kuczora, Miklos Csanady, and Barna Zsombor Klara.
>
>
> Bugs: HIVE-16334
> https://issues.apache.org/jira/browse/HIVE-16334
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> The patch contains the following changes:
> - Added a new config variable which defines the maximum length of the query string in the lock object
> - When constructing the HiveLockObject gets the configuration as a new parameter
> - When constructing the HiveLockObject truncates the length of the query string if it is necessary
> - Updates every occurance of the HiveLockObject creation
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e12fea1
> ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DummyTxnManager.java 53ee9c8
> ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockObject.java fff03df
> ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveTxnManagerImpl.java a371a5a
> ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDummyTxnManager.java de3b8ad
> ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestEmbeddedLockManager.java 0afbc1c
> ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestHiveLockObject.java 19cb129
> ql/src/test/org/apache/hadoop/hive/ql/lockmgr/zookeeper/TestZookeeperLockManager.java 3f9926e
>
>
> Diff: https://reviews.apache.org/r/58086/diff/1/
>
>
> Testing
> -------
>
> Added new unit test to test the truncation
> Updated the existing unit tests
>
>
> Thanks,
>
> Peter Vary
>
>