You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Zoltan Chovan via Review Board <no...@reviews.apache.org> on 2020/02/11 13:25:50 UTC
Review Request 72112: HIVE-22869 - Add locking benchmark to
metastore-tools/metastore-benchmarks
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72112/
-----------------------------------------------------------
Review request for hive, Denys Kuzmenko and Aron Hamvas.
Repository: hive-git
Description
-------
Add the possibility to run benchmarks on opening lock in the HMS. Currently this change only introduces single-threaded/single client testing. I'm planning to add multi-client support in a separate change.
Example parametrisation is as follows:
hbench -M "lock" -N 10 -d hive_test -W 0 -L 100
This will create N number (10) of tables to lock and it'll execute the lock() for L number (100) of times.
Diffs
-----
standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkTool.java 041cd76234
standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java f53f2ef43b
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java 7cc1e42a8b
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Util.java 101d6759c5
Diff: https://reviews.apache.org/r/72112/diff/1/
Testing
-------
Thanks,
Zoltan Chovan
Re: Review Request 72112: HIVE-22869 - Add locking benchmark to
metastore-tools/metastore-benchmarks
Posted by Denys Kuzmenko via Review Board <no...@reviews.apache.org>.
> On April 3, 2020, 11:28 a.m., Denys Kuzmenko wrote:
> > standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java
> > Lines 414 (patched)
> > <https://reviews.apache.org/r/72112/diff/1/?file=2210962#file2210962line429>
> >
> > I think more realistic scenario is having up to 10 tables with 10-1000 (configurable) partitions and locking not only on table level but partition.
>
> Zoltan Chovan wrote:
> I agree with that, however from a strictly api-testing viewpoint I'm not sure if there's any practical difference between creating locks for 10-1000 tables or 10-1000 partitions. The lock objects would just point to a different object, the number of HMS calls should be the same.
>
> Denys Kuzmenko wrote:
> what I ment, we should have a combination of table level and partition level locks (i.e. table level lock would join with blocking partition level locks and other way around)
ignore above if we cover here only read-locks
- Denys
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72112/#review220213
-----------------------------------------------------------
On April 9, 2020, 2:58 p.m., Zoltan Chovan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72112/
> -----------------------------------------------------------
>
> (Updated April 9, 2020, 2:58 p.m.)
>
>
> Review request for hive, Denys Kuzmenko, Aron Hamvas, Marton Bod, and Peter Vary.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Add the possibility to run benchmarks on opening lock in the HMS. Currently this change only introduces single-threaded/single client testing. I'm planning to add multi-client support in a separate change.
>
> Example parametrisation is as follows:
> hbench -M "lock" -N 10 -d hive_test -W 0 -L 100
> hbench -M ".*Lock.*" -N 10 -d hive_test -W 0 -L 100 -T 8 --params 100
>
> This will create N number (10) of tables to lock and it'll execute the lock() for L number (100) of times on T (8) threads where each thread will strart an HMS client
>
>
> Diffs
> -----
>
> standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkTool.java 041cd76234
> standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java f53f2ef43b
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java 7cc1e42a8b
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Util.java 101d6759c5
>
>
> Diff: https://reviews.apache.org/r/72112/diff/1/
>
>
> Testing
> -------
>
>
> File Attachments
> ----------------
>
> HIVE-22869.2.patch
> https://reviews.apache.org/media/uploaded/files/2020/04/02/5e35e835-f383-495f-9964-e66773fd6a90__HIVE-22869.2.patch
> HIVE-22869.3.patch
> https://reviews.apache.org/media/uploaded/files/2020/04/09/458beaa7-4743-40fb-a213-1ae4527be823__HIVE-22869.3.patch
>
>
> Thanks,
>
> Zoltan Chovan
>
>
Re: Review Request 72112: HIVE-22869 - Add locking benchmark to
metastore-tools/metastore-benchmarks
Posted by Denys Kuzmenko via Review Board <no...@reviews.apache.org>.
> On April 3, 2020, 11:28 a.m., Denys Kuzmenko wrote:
> > standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java
> > Lines 414 (patched)
> > <https://reviews.apache.org/r/72112/diff/1/?file=2210962#file2210962line429>
> >
> > I think more realistic scenario is having up to 10 tables with 10-1000 (configurable) partitions and locking not only on table level but partition.
>
> Zoltan Chovan wrote:
> I agree with that, however from a strictly api-testing viewpoint I'm not sure if there's any practical difference between creating locks for 10-1000 tables or 10-1000 partitions. The lock objects would just point to a different object, the number of HMS calls should be the same.
what I ment, we should have a combination of table level and partition level locks (i.e. table level lock would join with blocking partition level locks and other way around)
> On April 3, 2020, 11:28 a.m., Denys Kuzmenko wrote:
> > standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java
> > Lines 422 (patched)
> > <https://reviews.apache.org/r/72112/diff/1/?file=2210962#file2210962line437>
> >
> > I think, for more realistic scenario we should move commit inside of bench + add some sleep to simulate query execution. Currently locks number is just keep increasing and could go very high that shouldn't happen in real system.
>
> Zoltan Chovan wrote:
> If we start going down the query simulation road then this will be an e2e test, not an api benchmark :)
> Let's say we add sleep to simulate queries. If every query takes the same time (which is again not realistic) then we just add a wait and every client would try the calls at the same time again. If we add variable lenghts to quereis then this will not be a benchmark that is reproducable at will.
Could you please elaborate on your test scenarios?
If I am reading this right, currently we are only interested in a read-locks. Is it correct?
Also we are releasing locks almost instantaneously via commit, so we won't have big number of rows in HIVE_LOCKS table. Should we consider doing iniatial load of locks into this table?
- Denys
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72112/#review220213
-----------------------------------------------------------
On April 9, 2020, 2:58 p.m., Zoltan Chovan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72112/
> -----------------------------------------------------------
>
> (Updated April 9, 2020, 2:58 p.m.)
>
>
> Review request for hive, Denys Kuzmenko, Aron Hamvas, Marton Bod, and Peter Vary.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Add the possibility to run benchmarks on opening lock in the HMS. Currently this change only introduces single-threaded/single client testing. I'm planning to add multi-client support in a separate change.
>
> Example parametrisation is as follows:
> hbench -M "lock" -N 10 -d hive_test -W 0 -L 100
> hbench -M ".*Lock.*" -N 10 -d hive_test -W 0 -L 100 -T 8 --params 100
>
> This will create N number (10) of tables to lock and it'll execute the lock() for L number (100) of times on T (8) threads where each thread will strart an HMS client
>
>
> Diffs
> -----
>
> standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkTool.java 041cd76234
> standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java f53f2ef43b
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java 7cc1e42a8b
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Util.java 101d6759c5
>
>
> Diff: https://reviews.apache.org/r/72112/diff/1/
>
>
> Testing
> -------
>
>
> File Attachments
> ----------------
>
> HIVE-22869.2.patch
> https://reviews.apache.org/media/uploaded/files/2020/04/02/5e35e835-f383-495f-9964-e66773fd6a90__HIVE-22869.2.patch
> HIVE-22869.3.patch
> https://reviews.apache.org/media/uploaded/files/2020/04/09/458beaa7-4743-40fb-a213-1ae4527be823__HIVE-22869.3.patch
>
>
> Thanks,
>
> Zoltan Chovan
>
>
Re: Review Request 72112: HIVE-22869 - Add locking benchmark to
metastore-tools/metastore-benchmarks
Posted by Zoltan Chovan via Review Board <no...@reviews.apache.org>.
> On April 3, 2020, 11:28 a.m., Denys Kuzmenko wrote:
> > standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java
> > Lines 422 (patched)
> > <https://reviews.apache.org/r/72112/diff/1/?file=2210962#file2210962line437>
> >
> > I think, for more realistic scenario we should move commit inside of bench + add some sleep to simulate query execution. Currently locks number is just keep increasing and could go very high that shouldn't happen in real system.
If we start going down the query simulation road then this will be an e2e test, not an api benchmark :)
Let's say we add sleep to simulate queries. If every query takes the same time (which is again not realistic) then we just add a wait and every client would try the calls at the same time again. If we add variable lenghts to quereis then this will not be a benchmark that is reproducable at will.
- Zoltan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72112/#review220213
-----------------------------------------------------------
On April 9, 2020, 2:58 p.m., Zoltan Chovan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72112/
> -----------------------------------------------------------
>
> (Updated April 9, 2020, 2:58 p.m.)
>
>
> Review request for hive, Denys Kuzmenko, Aron Hamvas, Marton Bod, and Peter Vary.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Add the possibility to run benchmarks on opening lock in the HMS. Currently this change only introduces single-threaded/single client testing. I'm planning to add multi-client support in a separate change.
>
> Example parametrisation is as follows:
> hbench -M "lock" -N 10 -d hive_test -W 0 -L 100
> hbench -M ".*Lock.*" -N 10 -d hive_test -W 0 -L 100 -T 8 --params 100
>
> This will create N number (10) of tables to lock and it'll execute the lock() for L number (100) of times on T (8) threads where each thread will strart an HMS client
>
>
> Diffs
> -----
>
> standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkTool.java 041cd76234
> standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java f53f2ef43b
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java 7cc1e42a8b
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Util.java 101d6759c5
>
>
> Diff: https://reviews.apache.org/r/72112/diff/1/
>
>
> Testing
> -------
>
>
> File Attachments
> ----------------
>
> HIVE-22869.2.patch
> https://reviews.apache.org/media/uploaded/files/2020/04/02/5e35e835-f383-495f-9964-e66773fd6a90__HIVE-22869.2.patch
> HIVE-22869.3.patch
> https://reviews.apache.org/media/uploaded/files/2020/04/09/458beaa7-4743-40fb-a213-1ae4527be823__HIVE-22869.3.patch
>
>
> Thanks,
>
> Zoltan Chovan
>
>
Re: Review Request 72112: HIVE-22869 - Add locking benchmark to
metastore-tools/metastore-benchmarks
Posted by Zoltan Chovan via Review Board <no...@reviews.apache.org>.
> On April 3, 2020, 11:28 a.m., Denys Kuzmenko wrote:
> > standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java
> > Lines 410 (patched)
> > <https://reviews.apache.org/r/72112/diff/1/?file=2210962#file2210962line425>
> >
> > Do you really need to have physical tables?
no we don't :) removed this part
> On April 3, 2020, 11:28 a.m., Denys Kuzmenko wrote:
> > standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java
> > Lines 455 (patched)
> > <https://reviews.apache.org/r/72112/diff/1/?file=2210963#file2210963line464>
> >
> > Should we care about replPolicy here?
probably not, removed it
- Zoltan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72112/#review220213
-----------------------------------------------------------
On April 2, 2020, 2:13 p.m., Zoltan Chovan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72112/
> -----------------------------------------------------------
>
> (Updated April 2, 2020, 2:13 p.m.)
>
>
> Review request for hive, Denys Kuzmenko, Aron Hamvas, Marton Bod, and Peter Vary.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Add the possibility to run benchmarks on opening lock in the HMS. Currently this change only introduces single-threaded/single client testing. I'm planning to add multi-client support in a separate change.
>
> Example parametrisation is as follows:
> hbench -M "lock" -N 10 -d hive_test -W 0 -L 100
> hbench -M ".*Lock.*" -N 10 -d hive_test -W 0 -L 100 -T 8 --params 100
>
> This will create N number (10) of tables to lock and it'll execute the lock() for L number (100) of times on T (8) threads where each thread will strart an HMS client
>
>
> Diffs
> -----
>
> standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkTool.java 041cd76234
> standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java f53f2ef43b
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java 7cc1e42a8b
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Util.java 101d6759c5
>
>
> Diff: https://reviews.apache.org/r/72112/diff/1/
>
>
> Testing
> -------
>
>
> File Attachments
> ----------------
>
> HIVE-22869.2.patch
> https://reviews.apache.org/media/uploaded/files/2020/04/02/5e35e835-f383-495f-9964-e66773fd6a90__HIVE-22869.2.patch
>
>
> Thanks,
>
> Zoltan Chovan
>
>
Re: Review Request 72112: HIVE-22869 - Add locking benchmark to
metastore-tools/metastore-benchmarks
Posted by Zoltan Chovan via Review Board <no...@reviews.apache.org>.
> On April 3, 2020, 11:28 a.m., Denys Kuzmenko wrote:
> > standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java
> > Lines 414 (patched)
> > <https://reviews.apache.org/r/72112/diff/1/?file=2210962#file2210962line429>
> >
> > I think more realistic scenario is having up to 10 tables with 10-1000 (configurable) partitions and locking not only on table level but partition.
I agree with that, however from a strictly api-testing viewpoint I'm not sure if there's any practical difference between creating locks for 10-1000 tables or 10-1000 partitions. The lock objects would just point to a different object, the number of HMS calls should be the same.
- Zoltan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72112/#review220213
-----------------------------------------------------------
On April 9, 2020, 2:58 p.m., Zoltan Chovan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72112/
> -----------------------------------------------------------
>
> (Updated April 9, 2020, 2:58 p.m.)
>
>
> Review request for hive, Denys Kuzmenko, Aron Hamvas, Marton Bod, and Peter Vary.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Add the possibility to run benchmarks on opening lock in the HMS. Currently this change only introduces single-threaded/single client testing. I'm planning to add multi-client support in a separate change.
>
> Example parametrisation is as follows:
> hbench -M "lock" -N 10 -d hive_test -W 0 -L 100
> hbench -M ".*Lock.*" -N 10 -d hive_test -W 0 -L 100 -T 8 --params 100
>
> This will create N number (10) of tables to lock and it'll execute the lock() for L number (100) of times on T (8) threads where each thread will strart an HMS client
>
>
> Diffs
> -----
>
> standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkTool.java 041cd76234
> standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java f53f2ef43b
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java 7cc1e42a8b
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Util.java 101d6759c5
>
>
> Diff: https://reviews.apache.org/r/72112/diff/1/
>
>
> Testing
> -------
>
>
> File Attachments
> ----------------
>
> HIVE-22869.2.patch
> https://reviews.apache.org/media/uploaded/files/2020/04/02/5e35e835-f383-495f-9964-e66773fd6a90__HIVE-22869.2.patch
> HIVE-22869.3.patch
> https://reviews.apache.org/media/uploaded/files/2020/04/09/458beaa7-4743-40fb-a213-1ae4527be823__HIVE-22869.3.patch
>
>
> Thanks,
>
> Zoltan Chovan
>
>
Re: Review Request 72112: HIVE-22869 - Add locking benchmark to
metastore-tools/metastore-benchmarks
Posted by Denys Kuzmenko via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72112/#review220213
-----------------------------------------------------------
standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java
Lines 410 (patched)
<https://reviews.apache.org/r/72112/#comment308527>
Do you really need to have physical tables?
standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java
Lines 414 (patched)
<https://reviews.apache.org/r/72112/#comment308526>
I think more realistic scenario is having up to 10 tables with 10-1000 (configurable) partitions and locking not only on table level but partition.
standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java
Lines 422 (patched)
<https://reviews.apache.org/r/72112/#comment308529>
I think, for more realistic scenario we should move commit inside of bench + add some sleep to simulate query execution. Currently locks number is just keep increasing and could go very high that shouldn't happen in real system.
standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java
Lines 467 (patched)
<https://reviews.apache.org/r/72112/#comment308531>
This lock builder looks overcomplicated. To create Lock request it's enough to :
LockRequest req = new LockRequest(lockComponents, user, InetAddress.getLocalHost().getHostName());
req.setTxnid(txnId);
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java
Lines 455 (patched)
<https://reviews.apache.org/r/72112/#comment308530>
Should we care about replPolicy here?
- Denys Kuzmenko
On April 2, 2020, 2:13 p.m., Zoltan Chovan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72112/
> -----------------------------------------------------------
>
> (Updated April 2, 2020, 2:13 p.m.)
>
>
> Review request for hive, Denys Kuzmenko, Aron Hamvas, Marton Bod, and Peter Vary.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Add the possibility to run benchmarks on opening lock in the HMS. Currently this change only introduces single-threaded/single client testing. I'm planning to add multi-client support in a separate change.
>
> Example parametrisation is as follows:
> hbench -M "lock" -N 10 -d hive_test -W 0 -L 100
> hbench -M ".*Lock.*" -N 10 -d hive_test -W 0 -L 100 -T 8 --params 100
>
> This will create N number (10) of tables to lock and it'll execute the lock() for L number (100) of times on T (8) threads where each thread will strart an HMS client
>
>
> Diffs
> -----
>
> standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkTool.java 041cd76234
> standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java f53f2ef43b
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java 7cc1e42a8b
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Util.java 101d6759c5
>
>
> Diff: https://reviews.apache.org/r/72112/diff/1/
>
>
> Testing
> -------
>
>
> File Attachments
> ----------------
>
> HIVE-22869.2.patch
> https://reviews.apache.org/media/uploaded/files/2020/04/02/5e35e835-f383-495f-9964-e66773fd6a90__HIVE-22869.2.patch
>
>
> Thanks,
>
> Zoltan Chovan
>
>
Re: Review Request 72112: HIVE-22869 - Add locking benchmark to
metastore-tools/metastore-benchmarks
Posted by Denys Kuzmenko via Review Board <no...@reviews.apache.org>.
> On May 19, 2020, 7:31 a.m., Denys Kuzmenko wrote:
> > Ship It!
Zoli, I think you mentioned that hms-benchmark doesn't support running tests in multiple threads. Have you tried:
-T, --threads=<nThreads> number of concurrent threads
- Denys
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72112/#review220820
-----------------------------------------------------------
On May 18, 2020, 4:59 p.m., Zoltan Chovan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72112/
> -----------------------------------------------------------
>
> (Updated May 18, 2020, 4:59 p.m.)
>
>
> Review request for hive, Denys Kuzmenko, Aron Hamvas, Marton Bod, and Peter Vary.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Add the possibility to run benchmarks on opening lock in the HMS. Currently this change only introduces single-threaded/single client testing. I'm planning to add multi-client support in a separate change.
>
> Example parametrisation is as follows:
> hbench -M ".*Lock.*" -N 10 -d hive_test --params 10 --params 100 -d hive_test
>
> This will create N number (10) of locks for first --params number of tables (10) with second --params number of partitions (100) on T (8) threads where each thread will strart an HMS client and it'll use -d (hive_test) database;
>
>
> Diffs
> -----
>
> standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkTool.java 2ab9388301
> standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java d80c290b60
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkSuite.java 5211082a7d
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java 4e75edeae6
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Util.java 101d6759c5
>
>
> Diff: https://reviews.apache.org/r/72112/diff/2/
>
>
> Testing
> -------
>
>
> File Attachments
> ----------------
>
> HIVE-22869.2.patch
> https://reviews.apache.org/media/uploaded/files/2020/04/02/5e35e835-f383-495f-9964-e66773fd6a90__HIVE-22869.2.patch
> HIVE-22869.3.patch
> https://reviews.apache.org/media/uploaded/files/2020/04/09/458beaa7-4743-40fb-a213-1ae4527be823__HIVE-22869.3.patch
> HIVE-22869.4.patch
> https://reviews.apache.org/media/uploaded/files/2020/04/23/423c45d7-911e-4dd2-80b8-c6d3ad90633c__HIVE-22869.4.patch
> HIVE-22869.5.patch
> https://reviews.apache.org/media/uploaded/files/2020/05/12/a06f3b8c-f4ca-4067-a079-e0b6185266d4__HIVE-22869.5.patch
> HIVE-22869.6.patch
> https://reviews.apache.org/media/uploaded/files/2020/05/15/01254e94-1a8d-496d-ab31-628bd5584193__HIVE-22869.6.patch
> HIVE-22869.7.patch
> https://reviews.apache.org/media/uploaded/files/2020/05/18/343e8c4e-1f7c-4638-849a-15b448bc2515__HIVE-22869.7.patch
>
>
> Thanks,
>
> Zoltan Chovan
>
>
Re: Review Request 72112: HIVE-22869 - Add locking benchmark to
metastore-tools/metastore-benchmarks
Posted by Denys Kuzmenko via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72112/#review220820
-----------------------------------------------------------
Ship it!
Ship It!
- Denys Kuzmenko
On May 18, 2020, 4:59 p.m., Zoltan Chovan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72112/
> -----------------------------------------------------------
>
> (Updated May 18, 2020, 4:59 p.m.)
>
>
> Review request for hive, Denys Kuzmenko, Aron Hamvas, Marton Bod, and Peter Vary.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Add the possibility to run benchmarks on opening lock in the HMS. Currently this change only introduces single-threaded/single client testing. I'm planning to add multi-client support in a separate change.
>
> Example parametrisation is as follows:
> hbench -M ".*Lock.*" -N 10 -d hive_test --params 10 --params 100 -d hive_test
>
> This will create N number (10) of locks for first --params number of tables (10) with second --params number of partitions (100) on T (8) threads where each thread will strart an HMS client and it'll use -d (hive_test) database;
>
>
> Diffs
> -----
>
> standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkTool.java 2ab9388301
> standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java d80c290b60
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkSuite.java 5211082a7d
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java 4e75edeae6
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Util.java 101d6759c5
>
>
> Diff: https://reviews.apache.org/r/72112/diff/2/
>
>
> Testing
> -------
>
>
> File Attachments
> ----------------
>
> HIVE-22869.2.patch
> https://reviews.apache.org/media/uploaded/files/2020/04/02/5e35e835-f383-495f-9964-e66773fd6a90__HIVE-22869.2.patch
> HIVE-22869.3.patch
> https://reviews.apache.org/media/uploaded/files/2020/04/09/458beaa7-4743-40fb-a213-1ae4527be823__HIVE-22869.3.patch
> HIVE-22869.4.patch
> https://reviews.apache.org/media/uploaded/files/2020/04/23/423c45d7-911e-4dd2-80b8-c6d3ad90633c__HIVE-22869.4.patch
> HIVE-22869.5.patch
> https://reviews.apache.org/media/uploaded/files/2020/05/12/a06f3b8c-f4ca-4067-a079-e0b6185266d4__HIVE-22869.5.patch
> HIVE-22869.6.patch
> https://reviews.apache.org/media/uploaded/files/2020/05/15/01254e94-1a8d-496d-ab31-628bd5584193__HIVE-22869.6.patch
> HIVE-22869.7.patch
> https://reviews.apache.org/media/uploaded/files/2020/05/18/343e8c4e-1f7c-4638-849a-15b448bc2515__HIVE-22869.7.patch
>
>
> Thanks,
>
> Zoltan Chovan
>
>
Re: Review Request 72112: HIVE-22869 - Add locking benchmark to
metastore-tools/metastore-benchmarks
Posted by Denys Kuzmenko via Review Board <no...@reviews.apache.org>.
> On May 19, 2020, 7:28 a.m., Denys Kuzmenko wrote:
> > standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Util.java
> > Line 24 (original), 26 (patched)
> > <https://reviews.apache.org/r/72112/diff/2/?file=2231453#file2231453line26>
> >
> > I can still see wildcard import
however it's fixed in a latest patch.
- Denys
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72112/#review220818
-----------------------------------------------------------
On May 18, 2020, 4:59 p.m., Zoltan Chovan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72112/
> -----------------------------------------------------------
>
> (Updated May 18, 2020, 4:59 p.m.)
>
>
> Review request for hive, Denys Kuzmenko, Aron Hamvas, Marton Bod, and Peter Vary.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Add the possibility to run benchmarks on opening lock in the HMS. Currently this change only introduces single-threaded/single client testing. I'm planning to add multi-client support in a separate change.
>
> Example parametrisation is as follows:
> hbench -M ".*Lock.*" -N 10 -d hive_test --params 10 --params 100 -d hive_test
>
> This will create N number (10) of locks for first --params number of tables (10) with second --params number of partitions (100) on T (8) threads where each thread will strart an HMS client and it'll use -d (hive_test) database;
>
>
> Diffs
> -----
>
> standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkTool.java 2ab9388301
> standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java d80c290b60
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkSuite.java 5211082a7d
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java 4e75edeae6
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Util.java 101d6759c5
>
>
> Diff: https://reviews.apache.org/r/72112/diff/2/
>
>
> Testing
> -------
>
>
> File Attachments
> ----------------
>
> HIVE-22869.2.patch
> https://reviews.apache.org/media/uploaded/files/2020/04/02/5e35e835-f383-495f-9964-e66773fd6a90__HIVE-22869.2.patch
> HIVE-22869.3.patch
> https://reviews.apache.org/media/uploaded/files/2020/04/09/458beaa7-4743-40fb-a213-1ae4527be823__HIVE-22869.3.patch
> HIVE-22869.4.patch
> https://reviews.apache.org/media/uploaded/files/2020/04/23/423c45d7-911e-4dd2-80b8-c6d3ad90633c__HIVE-22869.4.patch
> HIVE-22869.5.patch
> https://reviews.apache.org/media/uploaded/files/2020/05/12/a06f3b8c-f4ca-4067-a079-e0b6185266d4__HIVE-22869.5.patch
> HIVE-22869.6.patch
> https://reviews.apache.org/media/uploaded/files/2020/05/15/01254e94-1a8d-496d-ab31-628bd5584193__HIVE-22869.6.patch
> HIVE-22869.7.patch
> https://reviews.apache.org/media/uploaded/files/2020/05/18/343e8c4e-1f7c-4638-849a-15b448bc2515__HIVE-22869.7.patch
>
>
> Thanks,
>
> Zoltan Chovan
>
>
Re: Review Request 72112: HIVE-22869 - Add locking benchmark to
metastore-tools/metastore-benchmarks
Posted by Denys Kuzmenko via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72112/#review220818
-----------------------------------------------------------
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Util.java
Line 24 (original), 26 (patched)
<https://reviews.apache.org/r/72112/#comment309529>
I can still see wildcard import
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Util.java
Line 40 (original), 37 (patched)
<https://reviews.apache.org/r/72112/#comment309530>
same here
- Denys Kuzmenko
On May 18, 2020, 4:59 p.m., Zoltan Chovan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72112/
> -----------------------------------------------------------
>
> (Updated May 18, 2020, 4:59 p.m.)
>
>
> Review request for hive, Denys Kuzmenko, Aron Hamvas, Marton Bod, and Peter Vary.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Add the possibility to run benchmarks on opening lock in the HMS. Currently this change only introduces single-threaded/single client testing. I'm planning to add multi-client support in a separate change.
>
> Example parametrisation is as follows:
> hbench -M ".*Lock.*" -N 10 -d hive_test --params 10 --params 100 -d hive_test
>
> This will create N number (10) of locks for first --params number of tables (10) with second --params number of partitions (100) on T (8) threads where each thread will strart an HMS client and it'll use -d (hive_test) database;
>
>
> Diffs
> -----
>
> standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkTool.java 2ab9388301
> standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java d80c290b60
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkSuite.java 5211082a7d
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java 4e75edeae6
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Util.java 101d6759c5
>
>
> Diff: https://reviews.apache.org/r/72112/diff/2/
>
>
> Testing
> -------
>
>
> File Attachments
> ----------------
>
> HIVE-22869.2.patch
> https://reviews.apache.org/media/uploaded/files/2020/04/02/5e35e835-f383-495f-9964-e66773fd6a90__HIVE-22869.2.patch
> HIVE-22869.3.patch
> https://reviews.apache.org/media/uploaded/files/2020/04/09/458beaa7-4743-40fb-a213-1ae4527be823__HIVE-22869.3.patch
> HIVE-22869.4.patch
> https://reviews.apache.org/media/uploaded/files/2020/04/23/423c45d7-911e-4dd2-80b8-c6d3ad90633c__HIVE-22869.4.patch
> HIVE-22869.5.patch
> https://reviews.apache.org/media/uploaded/files/2020/05/12/a06f3b8c-f4ca-4067-a079-e0b6185266d4__HIVE-22869.5.patch
> HIVE-22869.6.patch
> https://reviews.apache.org/media/uploaded/files/2020/05/15/01254e94-1a8d-496d-ab31-628bd5584193__HIVE-22869.6.patch
> HIVE-22869.7.patch
> https://reviews.apache.org/media/uploaded/files/2020/05/18/343e8c4e-1f7c-4638-849a-15b448bc2515__HIVE-22869.7.patch
>
>
> Thanks,
>
> Zoltan Chovan
>
>
Re: Review Request 72112: HIVE-22869 - Add locking benchmark to
metastore-tools/metastore-benchmarks
Posted by Zoltan Chovan via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72112/
-----------------------------------------------------------
(Updated May 18, 2020, 4:59 p.m.)
Review request for hive, Denys Kuzmenko, Aron Hamvas, Marton Bod, and Peter Vary.
Repository: hive-git
Description
-------
Add the possibility to run benchmarks on opening lock in the HMS. Currently this change only introduces single-threaded/single client testing. I'm planning to add multi-client support in a separate change.
Example parametrisation is as follows:
hbench -M ".*Lock.*" -N 10 -d hive_test --params 10 --params 100 -d hive_test
This will create N number (10) of locks for first --params number of tables (10) with second --params number of partitions (100) on T (8) threads where each thread will strart an HMS client and it'll use -d (hive_test) database;
Diffs
-----
standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkTool.java 2ab9388301
standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java d80c290b60
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkSuite.java 5211082a7d
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java 4e75edeae6
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Util.java 101d6759c5
Diff: https://reviews.apache.org/r/72112/diff/2/
Testing
-------
File Attachments (updated)
----------------
HIVE-22869.2.patch
https://reviews.apache.org/media/uploaded/files/2020/04/02/5e35e835-f383-495f-9964-e66773fd6a90__HIVE-22869.2.patch
HIVE-22869.3.patch
https://reviews.apache.org/media/uploaded/files/2020/04/09/458beaa7-4743-40fb-a213-1ae4527be823__HIVE-22869.3.patch
HIVE-22869.4.patch
https://reviews.apache.org/media/uploaded/files/2020/04/23/423c45d7-911e-4dd2-80b8-c6d3ad90633c__HIVE-22869.4.patch
HIVE-22869.5.patch
https://reviews.apache.org/media/uploaded/files/2020/05/12/a06f3b8c-f4ca-4067-a079-e0b6185266d4__HIVE-22869.5.patch
HIVE-22869.6.patch
https://reviews.apache.org/media/uploaded/files/2020/05/15/01254e94-1a8d-496d-ab31-628bd5584193__HIVE-22869.6.patch
HIVE-22869.7.patch
https://reviews.apache.org/media/uploaded/files/2020/05/18/343e8c4e-1f7c-4638-849a-15b448bc2515__HIVE-22869.7.patch
Thanks,
Zoltan Chovan
Re: Review Request 72112: HIVE-22869 - Add locking benchmark to
metastore-tools/metastore-benchmarks
Posted by Zoltan Chovan via Review Board <no...@reviews.apache.org>.
> On May 18, 2020, 7:58 a.m., Denys Kuzmenko wrote:
> > standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkTool.java
> > Lines 308 (patched)
> > <https://reviews.apache.org/r/72112/diff/2/?file=2231449#file2231449line309>
> >
> > Could we have null here? summarisedBenchmarkSuite.getResult().get(name)
there's a null check for that 4 lines above :)
> On May 18, 2020, 7:58 a.m., Denys Kuzmenko wrote:
> > standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java
> > Lines 362 (patched)
> > <https://reviews.apache.org/r/72112/diff/2/?file=2231452#file2231452line373>
> >
> > I think howMany is always 1. Isn't it?
no, this will be determined by the -N parameter (also called instances) see BenchmarkTool line 269
- Zoltan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72112/#review220804
-----------------------------------------------------------
On May 15, 2020, 9:12 a.m., Zoltan Chovan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72112/
> -----------------------------------------------------------
>
> (Updated May 15, 2020, 9:12 a.m.)
>
>
> Review request for hive, Denys Kuzmenko, Aron Hamvas, Marton Bod, and Peter Vary.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Add the possibility to run benchmarks on opening lock in the HMS. Currently this change only introduces single-threaded/single client testing. I'm planning to add multi-client support in a separate change.
>
> Example parametrisation is as follows:
> hbench -M ".*Lock.*" -N 10 -d hive_test --params 10 --params 100 -d hive_test
>
> This will create N number (10) of locks for first --params number of tables (10) with second --params number of partitions (100) on T (8) threads where each thread will strart an HMS client and it'll use -d (hive_test) database;
>
>
> Diffs
> -----
>
> standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkTool.java 2ab9388301
> standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java d80c290b60
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkSuite.java 5211082a7d
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java 4e75edeae6
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Util.java 101d6759c5
>
>
> Diff: https://reviews.apache.org/r/72112/diff/2/
>
>
> Testing
> -------
>
>
> File Attachments
> ----------------
>
> HIVE-22869.2.patch
> https://reviews.apache.org/media/uploaded/files/2020/04/02/5e35e835-f383-495f-9964-e66773fd6a90__HIVE-22869.2.patch
> HIVE-22869.3.patch
> https://reviews.apache.org/media/uploaded/files/2020/04/09/458beaa7-4743-40fb-a213-1ae4527be823__HIVE-22869.3.patch
> HIVE-22869.4.patch
> https://reviews.apache.org/media/uploaded/files/2020/04/23/423c45d7-911e-4dd2-80b8-c6d3ad90633c__HIVE-22869.4.patch
> HIVE-22869.5.patch
> https://reviews.apache.org/media/uploaded/files/2020/05/12/a06f3b8c-f4ca-4067-a079-e0b6185266d4__HIVE-22869.5.patch
> HIVE-22869.6.patch
> https://reviews.apache.org/media/uploaded/files/2020/05/15/01254e94-1a8d-496d-ab31-628bd5584193__HIVE-22869.6.patch
>
>
> Thanks,
>
> Zoltan Chovan
>
>
Re: Review Request 72112: HIVE-22869 - Add locking benchmark to
metastore-tools/metastore-benchmarks
Posted by Denys Kuzmenko via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72112/#review220804
-----------------------------------------------------------
LGTM, some minor comments.
standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkTool.java
Lines 308 (patched)
<https://reviews.apache.org/r/72112/#comment309505>
Could we have null here? summarisedBenchmarkSuite.getResult().get(name)
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java
Lines 362 (patched)
<https://reviews.apache.org/r/72112/#comment309508>
I think howMany is always 1. Isn't it?
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Util.java
Line 24 (original), 26 (patched)
<https://reviews.apache.org/r/72112/#comment309506>
avoid wildcard import
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Util.java
Line 40 (original), 37 (patched)
<https://reviews.apache.org/r/72112/#comment309507>
wildcard import
- Denys Kuzmenko
On May 15, 2020, 9:12 a.m., Zoltan Chovan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72112/
> -----------------------------------------------------------
>
> (Updated May 15, 2020, 9:12 a.m.)
>
>
> Review request for hive, Denys Kuzmenko, Aron Hamvas, Marton Bod, and Peter Vary.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Add the possibility to run benchmarks on opening lock in the HMS. Currently this change only introduces single-threaded/single client testing. I'm planning to add multi-client support in a separate change.
>
> Example parametrisation is as follows:
> hbench -M ".*Lock.*" -N 10 -d hive_test --params 10 --params 100 -d hive_test
>
> This will create N number (10) of locks for first --params number of tables (10) with second --params number of partitions (100) on T (8) threads where each thread will strart an HMS client and it'll use -d (hive_test) database;
>
>
> Diffs
> -----
>
> standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkTool.java 2ab9388301
> standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java d80c290b60
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkSuite.java 5211082a7d
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java 4e75edeae6
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Util.java 101d6759c5
>
>
> Diff: https://reviews.apache.org/r/72112/diff/2/
>
>
> Testing
> -------
>
>
> File Attachments
> ----------------
>
> HIVE-22869.2.patch
> https://reviews.apache.org/media/uploaded/files/2020/04/02/5e35e835-f383-495f-9964-e66773fd6a90__HIVE-22869.2.patch
> HIVE-22869.3.patch
> https://reviews.apache.org/media/uploaded/files/2020/04/09/458beaa7-4743-40fb-a213-1ae4527be823__HIVE-22869.3.patch
> HIVE-22869.4.patch
> https://reviews.apache.org/media/uploaded/files/2020/04/23/423c45d7-911e-4dd2-80b8-c6d3ad90633c__HIVE-22869.4.patch
> HIVE-22869.5.patch
> https://reviews.apache.org/media/uploaded/files/2020/05/12/a06f3b8c-f4ca-4067-a079-e0b6185266d4__HIVE-22869.5.patch
> HIVE-22869.6.patch
> https://reviews.apache.org/media/uploaded/files/2020/05/15/01254e94-1a8d-496d-ab31-628bd5584193__HIVE-22869.6.patch
>
>
> Thanks,
>
> Zoltan Chovan
>
>
Re: Review Request 72112: HIVE-22869 - Add locking benchmark to
metastore-tools/metastore-benchmarks
Posted by Zoltan Chovan via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72112/
-----------------------------------------------------------
(Updated May 15, 2020, 9:12 a.m.)
Review request for hive, Denys Kuzmenko, Aron Hamvas, Marton Bod, and Peter Vary.
Repository: hive-git
Description
-------
Add the possibility to run benchmarks on opening lock in the HMS. Currently this change only introduces single-threaded/single client testing. I'm planning to add multi-client support in a separate change.
Example parametrisation is as follows:
hbench -M ".*Lock.*" -N 10 -d hive_test --params 10 --params 100 -d hive_test
This will create N number (10) of locks for first --params number of tables (10) with second --params number of partitions (100) on T (8) threads where each thread will strart an HMS client and it'll use -d (hive_test) database;
Diffs
-----
standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkTool.java 2ab9388301
standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java d80c290b60
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkSuite.java 5211082a7d
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java 4e75edeae6
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Util.java 101d6759c5
Diff: https://reviews.apache.org/r/72112/diff/2/
Testing
-------
File Attachments (updated)
----------------
HIVE-22869.2.patch
https://reviews.apache.org/media/uploaded/files/2020/04/02/5e35e835-f383-495f-9964-e66773fd6a90__HIVE-22869.2.patch
HIVE-22869.3.patch
https://reviews.apache.org/media/uploaded/files/2020/04/09/458beaa7-4743-40fb-a213-1ae4527be823__HIVE-22869.3.patch
HIVE-22869.4.patch
https://reviews.apache.org/media/uploaded/files/2020/04/23/423c45d7-911e-4dd2-80b8-c6d3ad90633c__HIVE-22869.4.patch
HIVE-22869.5.patch
https://reviews.apache.org/media/uploaded/files/2020/05/12/a06f3b8c-f4ca-4067-a079-e0b6185266d4__HIVE-22869.5.patch
HIVE-22869.6.patch
https://reviews.apache.org/media/uploaded/files/2020/05/15/01254e94-1a8d-496d-ab31-628bd5584193__HIVE-22869.6.patch
Thanks,
Zoltan Chovan
Re: Review Request 72112: HIVE-22869 - Add locking benchmark to
metastore-tools/metastore-benchmarks
Posted by Zoltan Chovan via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72112/
-----------------------------------------------------------
(Updated May 12, 2020, 11:38 a.m.)
Review request for hive, Denys Kuzmenko, Aron Hamvas, Marton Bod, and Peter Vary.
Repository: hive-git
Description
-------
Add the possibility to run benchmarks on opening lock in the HMS. Currently this change only introduces single-threaded/single client testing. I'm planning to add multi-client support in a separate change.
Example parametrisation is as follows:
hbench -M "lock" -N 10 -d hive_test -W 0 -L 100
hbench -M ".*Lock.*" -N 10 -d hive_test -W 0 -L 100 -T 8 --params 100
This will create N number (10) of tables to lock and it'll execute the lock() for L number (100) of times on T (8) threads where each thread will strart an HMS client
Diffs (updated)
-----
standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkTool.java 2ab9388301
standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java d80c290b60
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkSuite.java 5211082a7d
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java 4e75edeae6
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Util.java 101d6759c5
Diff: https://reviews.apache.org/r/72112/diff/2/
Changes: https://reviews.apache.org/r/72112/diff/1-2/
Testing
-------
File Attachments (updated)
----------------
HIVE-22869.2.patch
https://reviews.apache.org/media/uploaded/files/2020/04/02/5e35e835-f383-495f-9964-e66773fd6a90__HIVE-22869.2.patch
HIVE-22869.3.patch
https://reviews.apache.org/media/uploaded/files/2020/04/09/458beaa7-4743-40fb-a213-1ae4527be823__HIVE-22869.3.patch
HIVE-22869.4.patch
https://reviews.apache.org/media/uploaded/files/2020/04/23/423c45d7-911e-4dd2-80b8-c6d3ad90633c__HIVE-22869.4.patch
HIVE-22869.5.patch
https://reviews.apache.org/media/uploaded/files/2020/05/12/a06f3b8c-f4ca-4067-a079-e0b6185266d4__HIVE-22869.5.patch
Thanks,
Zoltan Chovan
Re: Review Request 72112: HIVE-22869 - Add locking benchmark to
metastore-tools/metastore-benchmarks
Posted by Zoltan Chovan via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72112/
-----------------------------------------------------------
(Updated April 9, 2020, 2:58 p.m.)
Review request for hive, Denys Kuzmenko, Aron Hamvas, Marton Bod, and Peter Vary.
Repository: hive-git
Description
-------
Add the possibility to run benchmarks on opening lock in the HMS. Currently this change only introduces single-threaded/single client testing. I'm planning to add multi-client support in a separate change.
Example parametrisation is as follows:
hbench -M "lock" -N 10 -d hive_test -W 0 -L 100
hbench -M ".*Lock.*" -N 10 -d hive_test -W 0 -L 100 -T 8 --params 100
This will create N number (10) of tables to lock and it'll execute the lock() for L number (100) of times on T (8) threads where each thread will strart an HMS client
Diffs
-----
standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkTool.java 041cd76234
standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java f53f2ef43b
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java 7cc1e42a8b
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Util.java 101d6759c5
Diff: https://reviews.apache.org/r/72112/diff/1/
Testing
-------
File Attachments (updated)
----------------
HIVE-22869.2.patch
https://reviews.apache.org/media/uploaded/files/2020/04/02/5e35e835-f383-495f-9964-e66773fd6a90__HIVE-22869.2.patch
HIVE-22869.3.patch
https://reviews.apache.org/media/uploaded/files/2020/04/09/458beaa7-4743-40fb-a213-1ae4527be823__HIVE-22869.3.patch
Thanks,
Zoltan Chovan
Re: Review Request 72112: HIVE-22869 - Add locking benchmark to
metastore-tools/metastore-benchmarks
Posted by Zoltan Chovan via Review Board <no...@reviews.apache.org>.
> On April 3, 2020, 9:59 a.m., Peter Vary wrote:
> > standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java
> > Lines 341 (patched)
> > <https://reviews.apache.org/r/72112/diff/1/?file=2210963#file2210963line350>
> >
> > I put this in the code with HIVE-23042:
> > boolean openTxn(int numTxns) throws TException {
> > client.open_txns(new OpenTxnRequest(numTxns, "Test", "Host"));
> > return true;
> > }
> >
> > Maybe merge those?
The main difference between our two implementations of openTxn is that mine automatically returns the opened txn's id, in your version there has to be an additional getOpenTxns() call made to get the Id.
Not sure if getOpenTxns would return some other ids that belong to an other client when multiple threads are used, o sI might be misunderstanding the getOpenTxns() call.
What do you think?
- Zoltan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72112/#review220212
-----------------------------------------------------------
On April 2, 2020, 2:13 p.m., Zoltan Chovan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72112/
> -----------------------------------------------------------
>
> (Updated April 2, 2020, 2:13 p.m.)
>
>
> Review request for hive, Denys Kuzmenko, Aron Hamvas, Marton Bod, and Peter Vary.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Add the possibility to run benchmarks on opening lock in the HMS. Currently this change only introduces single-threaded/single client testing. I'm planning to add multi-client support in a separate change.
>
> Example parametrisation is as follows:
> hbench -M "lock" -N 10 -d hive_test -W 0 -L 100
> hbench -M ".*Lock.*" -N 10 -d hive_test -W 0 -L 100 -T 8 --params 100
>
> This will create N number (10) of tables to lock and it'll execute the lock() for L number (100) of times on T (8) threads where each thread will strart an HMS client
>
>
> Diffs
> -----
>
> standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkTool.java 041cd76234
> standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java f53f2ef43b
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java 7cc1e42a8b
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Util.java 101d6759c5
>
>
> Diff: https://reviews.apache.org/r/72112/diff/1/
>
>
> Testing
> -------
>
>
> File Attachments
> ----------------
>
> HIVE-22869.2.patch
> https://reviews.apache.org/media/uploaded/files/2020/04/02/5e35e835-f383-495f-9964-e66773fd6a90__HIVE-22869.2.patch
>
>
> Thanks,
>
> Zoltan Chovan
>
>
Re: Review Request 72112: HIVE-22869 - Add locking benchmark to
metastore-tools/metastore-benchmarks
Posted by Zoltan Chovan via Review Board <no...@reviews.apache.org>.
> On April 3, 2020, 9:59 a.m., Peter Vary wrote:
> > standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java
> > Lines 430 (patched)
> > <https://reviews.apache.org/r/72112/diff/1/?file=2210962#file2210962line445>
> >
> > Do we need this? Shouldn't the lock be unlocked by commitTnx?
you're right
- Zoltan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72112/#review220212
-----------------------------------------------------------
On April 2, 2020, 2:13 p.m., Zoltan Chovan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72112/
> -----------------------------------------------------------
>
> (Updated April 2, 2020, 2:13 p.m.)
>
>
> Review request for hive, Denys Kuzmenko, Aron Hamvas, Marton Bod, and Peter Vary.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Add the possibility to run benchmarks on opening lock in the HMS. Currently this change only introduces single-threaded/single client testing. I'm planning to add multi-client support in a separate change.
>
> Example parametrisation is as follows:
> hbench -M "lock" -N 10 -d hive_test -W 0 -L 100
> hbench -M ".*Lock.*" -N 10 -d hive_test -W 0 -L 100 -T 8 --params 100
>
> This will create N number (10) of tables to lock and it'll execute the lock() for L number (100) of times on T (8) threads where each thread will strart an HMS client
>
>
> Diffs
> -----
>
> standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkTool.java 041cd76234
> standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java f53f2ef43b
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java 7cc1e42a8b
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Util.java 101d6759c5
>
>
> Diff: https://reviews.apache.org/r/72112/diff/1/
>
>
> Testing
> -------
>
>
> File Attachments
> ----------------
>
> HIVE-22869.2.patch
> https://reviews.apache.org/media/uploaded/files/2020/04/02/5e35e835-f383-495f-9964-e66773fd6a90__HIVE-22869.2.patch
>
>
> Thanks,
>
> Zoltan Chovan
>
>
Re: Review Request 72112: HIVE-22869 - Add locking benchmark to
metastore-tools/metastore-benchmarks
Posted by Peter Vary via Review Board <no...@reviews.apache.org>.
> On ápr. 3, 2020, 9:59 de, Peter Vary wrote:
> > standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java
> > Lines 341 (patched)
> > <https://reviews.apache.org/r/72112/diff/1/?file=2210963#file2210963line350>
> >
> > I put this in the code with HIVE-23042:
> > boolean openTxn(int numTxns) throws TException {
> > client.open_txns(new OpenTxnRequest(numTxns, "Test", "Host"));
> > return true;
> > }
> >
> > Maybe merge those?
>
> Zoltan Chovan wrote:
> The main difference between our two implementations of openTxn is that mine automatically returns the opened txn's id, in your version there has to be an additional getOpenTxns() call made to get the Id.
> Not sure if getOpenTxns would return some other ids that belong to an other client when multiple threads are used, o sI might be misunderstanding the getOpenTxns() call.
> What do you think?
I still think it would be worth to keep only one version. In this specific case your new version, and use it on the other place.
What do you think?
- Peter
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72112/#review220212
-----------------------------------------------------------
On ápr. 9, 2020, 2:58 du, Zoltan Chovan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72112/
> -----------------------------------------------------------
>
> (Updated ápr. 9, 2020, 2:58 du)
>
>
> Review request for hive, Denys Kuzmenko, Aron Hamvas, Marton Bod, and Peter Vary.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Add the possibility to run benchmarks on opening lock in the HMS. Currently this change only introduces single-threaded/single client testing. I'm planning to add multi-client support in a separate change.
>
> Example parametrisation is as follows:
> hbench -M "lock" -N 10 -d hive_test -W 0 -L 100
> hbench -M ".*Lock.*" -N 10 -d hive_test -W 0 -L 100 -T 8 --params 100
>
> This will create N number (10) of tables to lock and it'll execute the lock() for L number (100) of times on T (8) threads where each thread will strart an HMS client
>
>
> Diffs
> -----
>
> standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkTool.java 041cd76234
> standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java f53f2ef43b
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java 7cc1e42a8b
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Util.java 101d6759c5
>
>
> Diff: https://reviews.apache.org/r/72112/diff/1/
>
>
> Testing
> -------
>
>
> File Attachments
> ----------------
>
> HIVE-22869.2.patch
> https://reviews.apache.org/media/uploaded/files/2020/04/02/5e35e835-f383-495f-9964-e66773fd6a90__HIVE-22869.2.patch
> HIVE-22869.3.patch
> https://reviews.apache.org/media/uploaded/files/2020/04/09/458beaa7-4743-40fb-a213-1ae4527be823__HIVE-22869.3.patch
>
>
> Thanks,
>
> Zoltan Chovan
>
>
Re: Review Request 72112: HIVE-22869 - Add locking benchmark to
metastore-tools/metastore-benchmarks
Posted by Peter Vary via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72112/#review220212
-----------------------------------------------------------
Thanks for the patch @Zoltan!
Some comments below.
Peter
standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java
Line 23 (original), 23 (patched)
<https://reviews.apache.org/r/72112/#comment308516>
Please avoid wildcard imports
standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java
Line 31 (original), 29 (patched)
<https://reviews.apache.org/r/72112/#comment308517>
Please avoid wildcard imports
standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java
Line 42 (original), 32 (patched)
<https://reviews.apache.org/r/72112/#comment308518>
Please avoid wildcard imports
standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java
Lines 423-425 (patched)
<https://reviews.apache.org/r/72112/#comment308519>
Maybe not here, maybe in another jira, but check, measure and execute all of the HMS calls for a typical query (not sure that the names, and a list is entirely correct:) ) :
- openTxn
- getOpenTxns
- getValidWriteIds
- enqueueLock
- commit/abort
standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java
Lines 430 (patched)
<https://reviews.apache.org/r/72112/#comment308520>
Do we need this? Shouldn't the lock be unlocked by commitTnx?
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java
Line 23 (original), 23 (patched)
<https://reviews.apache.org/r/72112/#comment308522>
wildcard import
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java
Line 51 (original), 44 (patched)
<https://reviews.apache.org/r/72112/#comment308521>
wildcard import
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java
Lines 341 (patched)
<https://reviews.apache.org/r/72112/#comment308523>
I put this in the code with HIVE-23042:
boolean openTxn(int numTxns) throws TException {
client.open_txns(new OpenTxnRequest(numTxns, "Test", "Host"));
return true;
}
Maybe merge those?
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Util.java
Line 24 (original), 24 (patched)
<https://reviews.apache.org/r/72112/#comment308524>
wildcard import
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Util.java
Line 40 (original), 35 (patched)
<https://reviews.apache.org/r/72112/#comment308525>
wildcard import
- Peter Vary
On ápr. 2, 2020, 2:13 du, Zoltan Chovan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72112/
> -----------------------------------------------------------
>
> (Updated ápr. 2, 2020, 2:13 du)
>
>
> Review request for hive, Denys Kuzmenko, Aron Hamvas, Marton Bod, and Peter Vary.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Add the possibility to run benchmarks on opening lock in the HMS. Currently this change only introduces single-threaded/single client testing. I'm planning to add multi-client support in a separate change.
>
> Example parametrisation is as follows:
> hbench -M "lock" -N 10 -d hive_test -W 0 -L 100
> hbench -M ".*Lock.*" -N 10 -d hive_test -W 0 -L 100 -T 8 --params 100
>
> This will create N number (10) of tables to lock and it'll execute the lock() for L number (100) of times on T (8) threads where each thread will strart an HMS client
>
>
> Diffs
> -----
>
> standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkTool.java 041cd76234
> standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java f53f2ef43b
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java 7cc1e42a8b
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Util.java 101d6759c5
>
>
> Diff: https://reviews.apache.org/r/72112/diff/1/
>
>
> Testing
> -------
>
>
> File Attachments
> ----------------
>
> HIVE-22869.2.patch
> https://reviews.apache.org/media/uploaded/files/2020/04/02/5e35e835-f383-495f-9964-e66773fd6a90__HIVE-22869.2.patch
>
>
> Thanks,
>
> Zoltan Chovan
>
>
Re: Review Request 72112: HIVE-22869 - Add locking benchmark to
metastore-tools/metastore-benchmarks
Posted by Zoltan Chovan via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72112/
-----------------------------------------------------------
(Updated April 2, 2020, 2:13 p.m.)
Review request for hive, Denys Kuzmenko and Aron Hamvas.
Repository: hive-git
Description (updated)
-------
Add the possibility to run benchmarks on opening lock in the HMS. Currently this change only introduces single-threaded/single client testing. I'm planning to add multi-client support in a separate change.
Example parametrisation is as follows:
hbench -M "lock" -N 10 -d hive_test -W 0 -L 100
hbench -M ".*Lock.*" -N 10 -d hive_test -W 0 -L 100 -T 8 --params 100
This will create N number (10) of tables to lock and it'll execute the lock() for L number (100) of times on T (8) threads where each thread will strart an HMS client
Diffs
-----
standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkTool.java 041cd76234
standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java f53f2ef43b
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java 7cc1e42a8b
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Util.java 101d6759c5
Diff: https://reviews.apache.org/r/72112/diff/1/
Testing
-------
File Attachments (updated)
----------------
HIVE-22869.2.patch
https://reviews.apache.org/media/uploaded/files/2020/04/02/5e35e835-f383-495f-9964-e66773fd6a90__HIVE-22869.2.patch
Thanks,
Zoltan Chovan