You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Zoltan Chovan via Review Board <no...@reviews.apache.org> on 2020/05/12 11:38:31 UTC
Re: Review Request 72112: HIVE-22869 - Add locking benchmark to
metastore-tools/metastore-benchmarks
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72112/
-----------------------------------------------------------
(Updated May 12, 2020, 11:38 a.m.)
Review request for hive, Denys Kuzmenko, Aron Hamvas, Marton Bod, and Peter Vary.
Repository: hive-git
Description
-------
Add the possibility to run benchmarks on opening lock in the HMS. Currently this change only introduces single-threaded/single client testing. I'm planning to add multi-client support in a separate change.
Example parametrisation is as follows:
hbench -M "lock" -N 10 -d hive_test -W 0 -L 100
hbench -M ".*Lock.*" -N 10 -d hive_test -W 0 -L 100 -T 8 --params 100
This will create N number (10) of tables to lock and it'll execute the lock() for L number (100) of times on T (8) threads where each thread will strart an HMS client
Diffs (updated)
-----
standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkTool.java 2ab9388301
standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java d80c290b60
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkSuite.java 5211082a7d
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java 4e75edeae6
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Util.java 101d6759c5
Diff: https://reviews.apache.org/r/72112/diff/2/
Changes: https://reviews.apache.org/r/72112/diff/1-2/
Testing
-------
File Attachments (updated)
----------------
HIVE-22869.2.patch
https://reviews.apache.org/media/uploaded/files/2020/04/02/5e35e835-f383-495f-9964-e66773fd6a90__HIVE-22869.2.patch
HIVE-22869.3.patch
https://reviews.apache.org/media/uploaded/files/2020/04/09/458beaa7-4743-40fb-a213-1ae4527be823__HIVE-22869.3.patch
HIVE-22869.4.patch
https://reviews.apache.org/media/uploaded/files/2020/04/23/423c45d7-911e-4dd2-80b8-c6d3ad90633c__HIVE-22869.4.patch
HIVE-22869.5.patch
https://reviews.apache.org/media/uploaded/files/2020/05/12/a06f3b8c-f4ca-4067-a079-e0b6185266d4__HIVE-22869.5.patch
Thanks,
Zoltan Chovan
Re: Review Request 72112: HIVE-22869 - Add locking benchmark to
metastore-tools/metastore-benchmarks
Posted by Denys Kuzmenko via Review Board <no...@reviews.apache.org>.
> On May 19, 2020, 7:31 a.m., Denys Kuzmenko wrote:
> > Ship It!
Zoli, I think you mentioned that hms-benchmark doesn't support running tests in multiple threads. Have you tried:
-T, --threads=<nThreads> number of concurrent threads
- Denys
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72112/#review220820
-----------------------------------------------------------
On May 18, 2020, 4:59 p.m., Zoltan Chovan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72112/
> -----------------------------------------------------------
>
> (Updated May 18, 2020, 4:59 p.m.)
>
>
> Review request for hive, Denys Kuzmenko, Aron Hamvas, Marton Bod, and Peter Vary.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Add the possibility to run benchmarks on opening lock in the HMS. Currently this change only introduces single-threaded/single client testing. I'm planning to add multi-client support in a separate change.
>
> Example parametrisation is as follows:
> hbench -M ".*Lock.*" -N 10 -d hive_test --params 10 --params 100 -d hive_test
>
> This will create N number (10) of locks for first --params number of tables (10) with second --params number of partitions (100) on T (8) threads where each thread will strart an HMS client and it'll use -d (hive_test) database;
>
>
> Diffs
> -----
>
> standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkTool.java 2ab9388301
> standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java d80c290b60
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkSuite.java 5211082a7d
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java 4e75edeae6
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Util.java 101d6759c5
>
>
> Diff: https://reviews.apache.org/r/72112/diff/2/
>
>
> Testing
> -------
>
>
> File Attachments
> ----------------
>
> HIVE-22869.2.patch
> https://reviews.apache.org/media/uploaded/files/2020/04/02/5e35e835-f383-495f-9964-e66773fd6a90__HIVE-22869.2.patch
> HIVE-22869.3.patch
> https://reviews.apache.org/media/uploaded/files/2020/04/09/458beaa7-4743-40fb-a213-1ae4527be823__HIVE-22869.3.patch
> HIVE-22869.4.patch
> https://reviews.apache.org/media/uploaded/files/2020/04/23/423c45d7-911e-4dd2-80b8-c6d3ad90633c__HIVE-22869.4.patch
> HIVE-22869.5.patch
> https://reviews.apache.org/media/uploaded/files/2020/05/12/a06f3b8c-f4ca-4067-a079-e0b6185266d4__HIVE-22869.5.patch
> HIVE-22869.6.patch
> https://reviews.apache.org/media/uploaded/files/2020/05/15/01254e94-1a8d-496d-ab31-628bd5584193__HIVE-22869.6.patch
> HIVE-22869.7.patch
> https://reviews.apache.org/media/uploaded/files/2020/05/18/343e8c4e-1f7c-4638-849a-15b448bc2515__HIVE-22869.7.patch
>
>
> Thanks,
>
> Zoltan Chovan
>
>
Re: Review Request 72112: HIVE-22869 - Add locking benchmark to
metastore-tools/metastore-benchmarks
Posted by Denys Kuzmenko via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72112/#review220820
-----------------------------------------------------------
Ship it!
Ship It!
- Denys Kuzmenko
On May 18, 2020, 4:59 p.m., Zoltan Chovan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72112/
> -----------------------------------------------------------
>
> (Updated May 18, 2020, 4:59 p.m.)
>
>
> Review request for hive, Denys Kuzmenko, Aron Hamvas, Marton Bod, and Peter Vary.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Add the possibility to run benchmarks on opening lock in the HMS. Currently this change only introduces single-threaded/single client testing. I'm planning to add multi-client support in a separate change.
>
> Example parametrisation is as follows:
> hbench -M ".*Lock.*" -N 10 -d hive_test --params 10 --params 100 -d hive_test
>
> This will create N number (10) of locks for first --params number of tables (10) with second --params number of partitions (100) on T (8) threads where each thread will strart an HMS client and it'll use -d (hive_test) database;
>
>
> Diffs
> -----
>
> standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkTool.java 2ab9388301
> standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java d80c290b60
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkSuite.java 5211082a7d
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java 4e75edeae6
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Util.java 101d6759c5
>
>
> Diff: https://reviews.apache.org/r/72112/diff/2/
>
>
> Testing
> -------
>
>
> File Attachments
> ----------------
>
> HIVE-22869.2.patch
> https://reviews.apache.org/media/uploaded/files/2020/04/02/5e35e835-f383-495f-9964-e66773fd6a90__HIVE-22869.2.patch
> HIVE-22869.3.patch
> https://reviews.apache.org/media/uploaded/files/2020/04/09/458beaa7-4743-40fb-a213-1ae4527be823__HIVE-22869.3.patch
> HIVE-22869.4.patch
> https://reviews.apache.org/media/uploaded/files/2020/04/23/423c45d7-911e-4dd2-80b8-c6d3ad90633c__HIVE-22869.4.patch
> HIVE-22869.5.patch
> https://reviews.apache.org/media/uploaded/files/2020/05/12/a06f3b8c-f4ca-4067-a079-e0b6185266d4__HIVE-22869.5.patch
> HIVE-22869.6.patch
> https://reviews.apache.org/media/uploaded/files/2020/05/15/01254e94-1a8d-496d-ab31-628bd5584193__HIVE-22869.6.patch
> HIVE-22869.7.patch
> https://reviews.apache.org/media/uploaded/files/2020/05/18/343e8c4e-1f7c-4638-849a-15b448bc2515__HIVE-22869.7.patch
>
>
> Thanks,
>
> Zoltan Chovan
>
>
Re: Review Request 72112: HIVE-22869 - Add locking benchmark to
metastore-tools/metastore-benchmarks
Posted by Denys Kuzmenko via Review Board <no...@reviews.apache.org>.
> On May 19, 2020, 7:28 a.m., Denys Kuzmenko wrote:
> > standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Util.java
> > Line 24 (original), 26 (patched)
> > <https://reviews.apache.org/r/72112/diff/2/?file=2231453#file2231453line26>
> >
> > I can still see wildcard import
however it's fixed in a latest patch.
- Denys
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72112/#review220818
-----------------------------------------------------------
On May 18, 2020, 4:59 p.m., Zoltan Chovan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72112/
> -----------------------------------------------------------
>
> (Updated May 18, 2020, 4:59 p.m.)
>
>
> Review request for hive, Denys Kuzmenko, Aron Hamvas, Marton Bod, and Peter Vary.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Add the possibility to run benchmarks on opening lock in the HMS. Currently this change only introduces single-threaded/single client testing. I'm planning to add multi-client support in a separate change.
>
> Example parametrisation is as follows:
> hbench -M ".*Lock.*" -N 10 -d hive_test --params 10 --params 100 -d hive_test
>
> This will create N number (10) of locks for first --params number of tables (10) with second --params number of partitions (100) on T (8) threads where each thread will strart an HMS client and it'll use -d (hive_test) database;
>
>
> Diffs
> -----
>
> standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkTool.java 2ab9388301
> standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java d80c290b60
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkSuite.java 5211082a7d
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java 4e75edeae6
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Util.java 101d6759c5
>
>
> Diff: https://reviews.apache.org/r/72112/diff/2/
>
>
> Testing
> -------
>
>
> File Attachments
> ----------------
>
> HIVE-22869.2.patch
> https://reviews.apache.org/media/uploaded/files/2020/04/02/5e35e835-f383-495f-9964-e66773fd6a90__HIVE-22869.2.patch
> HIVE-22869.3.patch
> https://reviews.apache.org/media/uploaded/files/2020/04/09/458beaa7-4743-40fb-a213-1ae4527be823__HIVE-22869.3.patch
> HIVE-22869.4.patch
> https://reviews.apache.org/media/uploaded/files/2020/04/23/423c45d7-911e-4dd2-80b8-c6d3ad90633c__HIVE-22869.4.patch
> HIVE-22869.5.patch
> https://reviews.apache.org/media/uploaded/files/2020/05/12/a06f3b8c-f4ca-4067-a079-e0b6185266d4__HIVE-22869.5.patch
> HIVE-22869.6.patch
> https://reviews.apache.org/media/uploaded/files/2020/05/15/01254e94-1a8d-496d-ab31-628bd5584193__HIVE-22869.6.patch
> HIVE-22869.7.patch
> https://reviews.apache.org/media/uploaded/files/2020/05/18/343e8c4e-1f7c-4638-849a-15b448bc2515__HIVE-22869.7.patch
>
>
> Thanks,
>
> Zoltan Chovan
>
>
Re: Review Request 72112: HIVE-22869 - Add locking benchmark to
metastore-tools/metastore-benchmarks
Posted by Denys Kuzmenko via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72112/#review220818
-----------------------------------------------------------
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Util.java
Line 24 (original), 26 (patched)
<https://reviews.apache.org/r/72112/#comment309529>
I can still see wildcard import
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Util.java
Line 40 (original), 37 (patched)
<https://reviews.apache.org/r/72112/#comment309530>
same here
- Denys Kuzmenko
On May 18, 2020, 4:59 p.m., Zoltan Chovan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72112/
> -----------------------------------------------------------
>
> (Updated May 18, 2020, 4:59 p.m.)
>
>
> Review request for hive, Denys Kuzmenko, Aron Hamvas, Marton Bod, and Peter Vary.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Add the possibility to run benchmarks on opening lock in the HMS. Currently this change only introduces single-threaded/single client testing. I'm planning to add multi-client support in a separate change.
>
> Example parametrisation is as follows:
> hbench -M ".*Lock.*" -N 10 -d hive_test --params 10 --params 100 -d hive_test
>
> This will create N number (10) of locks for first --params number of tables (10) with second --params number of partitions (100) on T (8) threads where each thread will strart an HMS client and it'll use -d (hive_test) database;
>
>
> Diffs
> -----
>
> standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkTool.java 2ab9388301
> standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java d80c290b60
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkSuite.java 5211082a7d
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java 4e75edeae6
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Util.java 101d6759c5
>
>
> Diff: https://reviews.apache.org/r/72112/diff/2/
>
>
> Testing
> -------
>
>
> File Attachments
> ----------------
>
> HIVE-22869.2.patch
> https://reviews.apache.org/media/uploaded/files/2020/04/02/5e35e835-f383-495f-9964-e66773fd6a90__HIVE-22869.2.patch
> HIVE-22869.3.patch
> https://reviews.apache.org/media/uploaded/files/2020/04/09/458beaa7-4743-40fb-a213-1ae4527be823__HIVE-22869.3.patch
> HIVE-22869.4.patch
> https://reviews.apache.org/media/uploaded/files/2020/04/23/423c45d7-911e-4dd2-80b8-c6d3ad90633c__HIVE-22869.4.patch
> HIVE-22869.5.patch
> https://reviews.apache.org/media/uploaded/files/2020/05/12/a06f3b8c-f4ca-4067-a079-e0b6185266d4__HIVE-22869.5.patch
> HIVE-22869.6.patch
> https://reviews.apache.org/media/uploaded/files/2020/05/15/01254e94-1a8d-496d-ab31-628bd5584193__HIVE-22869.6.patch
> HIVE-22869.7.patch
> https://reviews.apache.org/media/uploaded/files/2020/05/18/343e8c4e-1f7c-4638-849a-15b448bc2515__HIVE-22869.7.patch
>
>
> Thanks,
>
> Zoltan Chovan
>
>
Re: Review Request 72112: HIVE-22869 - Add locking benchmark to
metastore-tools/metastore-benchmarks
Posted by Zoltan Chovan via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72112/
-----------------------------------------------------------
(Updated May 18, 2020, 4:59 p.m.)
Review request for hive, Denys Kuzmenko, Aron Hamvas, Marton Bod, and Peter Vary.
Repository: hive-git
Description
-------
Add the possibility to run benchmarks on opening lock in the HMS. Currently this change only introduces single-threaded/single client testing. I'm planning to add multi-client support in a separate change.
Example parametrisation is as follows:
hbench -M ".*Lock.*" -N 10 -d hive_test --params 10 --params 100 -d hive_test
This will create N number (10) of locks for first --params number of tables (10) with second --params number of partitions (100) on T (8) threads where each thread will strart an HMS client and it'll use -d (hive_test) database;
Diffs
-----
standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkTool.java 2ab9388301
standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java d80c290b60
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkSuite.java 5211082a7d
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java 4e75edeae6
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Util.java 101d6759c5
Diff: https://reviews.apache.org/r/72112/diff/2/
Testing
-------
File Attachments (updated)
----------------
HIVE-22869.2.patch
https://reviews.apache.org/media/uploaded/files/2020/04/02/5e35e835-f383-495f-9964-e66773fd6a90__HIVE-22869.2.patch
HIVE-22869.3.patch
https://reviews.apache.org/media/uploaded/files/2020/04/09/458beaa7-4743-40fb-a213-1ae4527be823__HIVE-22869.3.patch
HIVE-22869.4.patch
https://reviews.apache.org/media/uploaded/files/2020/04/23/423c45d7-911e-4dd2-80b8-c6d3ad90633c__HIVE-22869.4.patch
HIVE-22869.5.patch
https://reviews.apache.org/media/uploaded/files/2020/05/12/a06f3b8c-f4ca-4067-a079-e0b6185266d4__HIVE-22869.5.patch
HIVE-22869.6.patch
https://reviews.apache.org/media/uploaded/files/2020/05/15/01254e94-1a8d-496d-ab31-628bd5584193__HIVE-22869.6.patch
HIVE-22869.7.patch
https://reviews.apache.org/media/uploaded/files/2020/05/18/343e8c4e-1f7c-4638-849a-15b448bc2515__HIVE-22869.7.patch
Thanks,
Zoltan Chovan
Re: Review Request 72112: HIVE-22869 - Add locking benchmark to
metastore-tools/metastore-benchmarks
Posted by Zoltan Chovan via Review Board <no...@reviews.apache.org>.
> On May 18, 2020, 7:58 a.m., Denys Kuzmenko wrote:
> > standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkTool.java
> > Lines 308 (patched)
> > <https://reviews.apache.org/r/72112/diff/2/?file=2231449#file2231449line309>
> >
> > Could we have null here? summarisedBenchmarkSuite.getResult().get(name)
there's a null check for that 4 lines above :)
> On May 18, 2020, 7:58 a.m., Denys Kuzmenko wrote:
> > standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java
> > Lines 362 (patched)
> > <https://reviews.apache.org/r/72112/diff/2/?file=2231452#file2231452line373>
> >
> > I think howMany is always 1. Isn't it?
no, this will be determined by the -N parameter (also called instances) see BenchmarkTool line 269
- Zoltan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72112/#review220804
-----------------------------------------------------------
On May 15, 2020, 9:12 a.m., Zoltan Chovan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72112/
> -----------------------------------------------------------
>
> (Updated May 15, 2020, 9:12 a.m.)
>
>
> Review request for hive, Denys Kuzmenko, Aron Hamvas, Marton Bod, and Peter Vary.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Add the possibility to run benchmarks on opening lock in the HMS. Currently this change only introduces single-threaded/single client testing. I'm planning to add multi-client support in a separate change.
>
> Example parametrisation is as follows:
> hbench -M ".*Lock.*" -N 10 -d hive_test --params 10 --params 100 -d hive_test
>
> This will create N number (10) of locks for first --params number of tables (10) with second --params number of partitions (100) on T (8) threads where each thread will strart an HMS client and it'll use -d (hive_test) database;
>
>
> Diffs
> -----
>
> standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkTool.java 2ab9388301
> standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java d80c290b60
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkSuite.java 5211082a7d
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java 4e75edeae6
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Util.java 101d6759c5
>
>
> Diff: https://reviews.apache.org/r/72112/diff/2/
>
>
> Testing
> -------
>
>
> File Attachments
> ----------------
>
> HIVE-22869.2.patch
> https://reviews.apache.org/media/uploaded/files/2020/04/02/5e35e835-f383-495f-9964-e66773fd6a90__HIVE-22869.2.patch
> HIVE-22869.3.patch
> https://reviews.apache.org/media/uploaded/files/2020/04/09/458beaa7-4743-40fb-a213-1ae4527be823__HIVE-22869.3.patch
> HIVE-22869.4.patch
> https://reviews.apache.org/media/uploaded/files/2020/04/23/423c45d7-911e-4dd2-80b8-c6d3ad90633c__HIVE-22869.4.patch
> HIVE-22869.5.patch
> https://reviews.apache.org/media/uploaded/files/2020/05/12/a06f3b8c-f4ca-4067-a079-e0b6185266d4__HIVE-22869.5.patch
> HIVE-22869.6.patch
> https://reviews.apache.org/media/uploaded/files/2020/05/15/01254e94-1a8d-496d-ab31-628bd5584193__HIVE-22869.6.patch
>
>
> Thanks,
>
> Zoltan Chovan
>
>
Re: Review Request 72112: HIVE-22869 - Add locking benchmark to
metastore-tools/metastore-benchmarks
Posted by Denys Kuzmenko via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72112/#review220804
-----------------------------------------------------------
LGTM, some minor comments.
standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkTool.java
Lines 308 (patched)
<https://reviews.apache.org/r/72112/#comment309505>
Could we have null here? summarisedBenchmarkSuite.getResult().get(name)
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java
Lines 362 (patched)
<https://reviews.apache.org/r/72112/#comment309508>
I think howMany is always 1. Isn't it?
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Util.java
Line 24 (original), 26 (patched)
<https://reviews.apache.org/r/72112/#comment309506>
avoid wildcard import
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Util.java
Line 40 (original), 37 (patched)
<https://reviews.apache.org/r/72112/#comment309507>
wildcard import
- Denys Kuzmenko
On May 15, 2020, 9:12 a.m., Zoltan Chovan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72112/
> -----------------------------------------------------------
>
> (Updated May 15, 2020, 9:12 a.m.)
>
>
> Review request for hive, Denys Kuzmenko, Aron Hamvas, Marton Bod, and Peter Vary.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Add the possibility to run benchmarks on opening lock in the HMS. Currently this change only introduces single-threaded/single client testing. I'm planning to add multi-client support in a separate change.
>
> Example parametrisation is as follows:
> hbench -M ".*Lock.*" -N 10 -d hive_test --params 10 --params 100 -d hive_test
>
> This will create N number (10) of locks for first --params number of tables (10) with second --params number of partitions (100) on T (8) threads where each thread will strart an HMS client and it'll use -d (hive_test) database;
>
>
> Diffs
> -----
>
> standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkTool.java 2ab9388301
> standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java d80c290b60
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkSuite.java 5211082a7d
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java 4e75edeae6
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Util.java 101d6759c5
>
>
> Diff: https://reviews.apache.org/r/72112/diff/2/
>
>
> Testing
> -------
>
>
> File Attachments
> ----------------
>
> HIVE-22869.2.patch
> https://reviews.apache.org/media/uploaded/files/2020/04/02/5e35e835-f383-495f-9964-e66773fd6a90__HIVE-22869.2.patch
> HIVE-22869.3.patch
> https://reviews.apache.org/media/uploaded/files/2020/04/09/458beaa7-4743-40fb-a213-1ae4527be823__HIVE-22869.3.patch
> HIVE-22869.4.patch
> https://reviews.apache.org/media/uploaded/files/2020/04/23/423c45d7-911e-4dd2-80b8-c6d3ad90633c__HIVE-22869.4.patch
> HIVE-22869.5.patch
> https://reviews.apache.org/media/uploaded/files/2020/05/12/a06f3b8c-f4ca-4067-a079-e0b6185266d4__HIVE-22869.5.patch
> HIVE-22869.6.patch
> https://reviews.apache.org/media/uploaded/files/2020/05/15/01254e94-1a8d-496d-ab31-628bd5584193__HIVE-22869.6.patch
>
>
> Thanks,
>
> Zoltan Chovan
>
>
Re: Review Request 72112: HIVE-22869 - Add locking benchmark to
metastore-tools/metastore-benchmarks
Posted by Zoltan Chovan via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72112/
-----------------------------------------------------------
(Updated May 15, 2020, 9:12 a.m.)
Review request for hive, Denys Kuzmenko, Aron Hamvas, Marton Bod, and Peter Vary.
Repository: hive-git
Description
-------
Add the possibility to run benchmarks on opening lock in the HMS. Currently this change only introduces single-threaded/single client testing. I'm planning to add multi-client support in a separate change.
Example parametrisation is as follows:
hbench -M ".*Lock.*" -N 10 -d hive_test --params 10 --params 100 -d hive_test
This will create N number (10) of locks for first --params number of tables (10) with second --params number of partitions (100) on T (8) threads where each thread will strart an HMS client and it'll use -d (hive_test) database;
Diffs
-----
standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkTool.java 2ab9388301
standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java d80c290b60
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkSuite.java 5211082a7d
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java 4e75edeae6
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Util.java 101d6759c5
Diff: https://reviews.apache.org/r/72112/diff/2/
Testing
-------
File Attachments (updated)
----------------
HIVE-22869.2.patch
https://reviews.apache.org/media/uploaded/files/2020/04/02/5e35e835-f383-495f-9964-e66773fd6a90__HIVE-22869.2.patch
HIVE-22869.3.patch
https://reviews.apache.org/media/uploaded/files/2020/04/09/458beaa7-4743-40fb-a213-1ae4527be823__HIVE-22869.3.patch
HIVE-22869.4.patch
https://reviews.apache.org/media/uploaded/files/2020/04/23/423c45d7-911e-4dd2-80b8-c6d3ad90633c__HIVE-22869.4.patch
HIVE-22869.5.patch
https://reviews.apache.org/media/uploaded/files/2020/05/12/a06f3b8c-f4ca-4067-a079-e0b6185266d4__HIVE-22869.5.patch
HIVE-22869.6.patch
https://reviews.apache.org/media/uploaded/files/2020/05/15/01254e94-1a8d-496d-ab31-628bd5584193__HIVE-22869.6.patch
Thanks,
Zoltan Chovan