You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Peter Vary via Review Board <no...@reviews.apache.org> on 2020/04/20 14:53:05 UTC
Review Request 72392: HIVE-23103 Oracle statement batching
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72392/
-----------------------------------------------------------
Review request for hive, Denys Kuzmenko and Marton Bod.
Bugs: HIVE-23103
https://issues.apache.org/jira/browse/HIVE-23103
Repository: hive-git
Description
-------
Examine how to really get better performance for oracle statement batches.
Oracle JDBC doc describes:
The Oracle implementation of standard update batching does not implement true batching for generic statements and callable statements. Even though Oracle JDBC supports the use of standard batching for Statement and CallableStatement objects, you are unlikely to see performance improvement.
I would look for connection properties to set, so it is handled anyway, or if not, then use:
begin
query1;
query2;
query3;
end;
to we will have only a single roundtrip for the db.
Diffs
-----
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnDbUtil.java bb29410e7d
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java d080df417b
Diff: https://reviews.apache.org/r/72392/diff/1/
Testing
-------
Baseline:
Benchmark (dbProduct) (txnType) Mode Cnt Score Error Units
TxnHandlerBenchRunner.commitTxn ORACLE DEFAULT ss 100 42.988 ± 4.569 ms/op
TxnHandlerBenchRunner.commitTxn ORACLE READ_ONLY ss 100 45.029 ± 4.686 ms/op
After patch:
Benchmark (dbProduct) (txnType) Mode Cnt Score Error Units
TxnHandlerBenchRunner.commitTxn ORACLE DEFAULT ss 100 36.208 ± 3.869 ms/op
TxnHandlerBenchRunner.commitTxn ORACLE READ_ONLY ss 100 37.038 ± 3.746 ms/op
Thanks,
Peter Vary
Re: Review Request 72392: HIVE-23103 Oracle statement batching
Posted by Peter Vary via Review Board <no...@reviews.apache.org>.
> On ápr. 21, 2020, 7:51 de, Marton Bod wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnDbUtil.java
> > Lines 662 (patched)
> > <https://reviews.apache.org/r/72392/diff/1/?file=2223854#file2223854line662>
> >
> > Once this part executes, we wouldn't have 'begin' for the next batch, no? Also, the sb would need to be cleared I think
good find!
Fixed it
- Peter
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72392/#review220387
-----------------------------------------------------------
On ápr. 21, 2020, 12:41 du, Peter Vary wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72392/
> -----------------------------------------------------------
>
> (Updated ápr. 21, 2020, 12:41 du)
>
>
> Review request for hive, Denys Kuzmenko and Marton Bod.
>
>
> Bugs: HIVE-23103
> https://issues.apache.org/jira/browse/HIVE-23103
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Examine how to really get better performance for oracle statement batches.
>
> Oracle JDBC doc describes:
>
> The Oracle implementation of standard update batching does not implement true batching for generic statements and callable statements. Even though Oracle JDBC supports the use of standard batching for Statement and CallableStatement objects, you are unlikely to see performance improvement.
>
> I would look for connection properties to set, so it is handled anyway, or if not, then use:
>
> begin
> query1;
> query2;
> query3;
> end;
> to we will have only a single roundtrip for the db.
>
>
> Diffs
> -----
>
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnDbUtil.java bb29410e7d
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java d080df417b
>
>
> Diff: https://reviews.apache.org/r/72392/diff/2/
>
>
> Testing
> -------
>
> Baseline:
> Benchmark (dbProduct) (txnType) Mode Cnt Score Error Units
> TxnHandlerBenchRunner.commitTxn ORACLE DEFAULT ss 100 42.988 ± 4.569 ms/op
> TxnHandlerBenchRunner.commitTxn ORACLE READ_ONLY ss 100 45.029 ± 4.686 ms/op
>
> After patch:
> Benchmark (dbProduct) (txnType) Mode Cnt Score Error Units
> TxnHandlerBenchRunner.commitTxn ORACLE DEFAULT ss 100 36.208 ± 3.869 ms/op
> TxnHandlerBenchRunner.commitTxn ORACLE READ_ONLY ss 100 37.038 ± 3.746 ms/op
>
>
> Thanks,
>
> Peter Vary
>
>
Re: Review Request 72392: HIVE-23103 Oracle statement batching
Posted by Marton Bod <ma...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72392/#review220387
-----------------------------------------------------------
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnDbUtil.java
Lines 662 (patched)
<https://reviews.apache.org/r/72392/#comment308749>
Once this part executes, we wouldn't have 'begin' for the next batch, no? Also, the sb would need to be cleared I think
- Marton Bod
On April 20, 2020, 2:53 p.m., Peter Vary wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72392/
> -----------------------------------------------------------
>
> (Updated April 20, 2020, 2:53 p.m.)
>
>
> Review request for hive, Denys Kuzmenko and Marton Bod.
>
>
> Bugs: HIVE-23103
> https://issues.apache.org/jira/browse/HIVE-23103
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Examine how to really get better performance for oracle statement batches.
>
> Oracle JDBC doc describes:
>
> The Oracle implementation of standard update batching does not implement true batching for generic statements and callable statements. Even though Oracle JDBC supports the use of standard batching for Statement and CallableStatement objects, you are unlikely to see performance improvement.
>
> I would look for connection properties to set, so it is handled anyway, or if not, then use:
>
> begin
> query1;
> query2;
> query3;
> end;
> to we will have only a single roundtrip for the db.
>
>
> Diffs
> -----
>
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnDbUtil.java bb29410e7d
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java d080df417b
>
>
> Diff: https://reviews.apache.org/r/72392/diff/1/
>
>
> Testing
> -------
>
> Baseline:
> Benchmark (dbProduct) (txnType) Mode Cnt Score Error Units
> TxnHandlerBenchRunner.commitTxn ORACLE DEFAULT ss 100 42.988 ± 4.569 ms/op
> TxnHandlerBenchRunner.commitTxn ORACLE READ_ONLY ss 100 45.029 ± 4.686 ms/op
>
> After patch:
> Benchmark (dbProduct) (txnType) Mode Cnt Score Error Units
> TxnHandlerBenchRunner.commitTxn ORACLE DEFAULT ss 100 36.208 ± 3.869 ms/op
> TxnHandlerBenchRunner.commitTxn ORACLE READ_ONLY ss 100 37.038 ± 3.746 ms/op
>
>
> Thanks,
>
> Peter Vary
>
>
Re: Review Request 72392: HIVE-23103 Oracle statement batching
Posted by Marton Bod <ma...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72392/#review220401
-----------------------------------------------------------
Ship it!
Ship It!
- Marton Bod
On April 21, 2020, 12:41 p.m., Peter Vary wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72392/
> -----------------------------------------------------------
>
> (Updated April 21, 2020, 12:41 p.m.)
>
>
> Review request for hive, Denys Kuzmenko and Marton Bod.
>
>
> Bugs: HIVE-23103
> https://issues.apache.org/jira/browse/HIVE-23103
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Examine how to really get better performance for oracle statement batches.
>
> Oracle JDBC doc describes:
>
> The Oracle implementation of standard update batching does not implement true batching for generic statements and callable statements. Even though Oracle JDBC supports the use of standard batching for Statement and CallableStatement objects, you are unlikely to see performance improvement.
>
> I would look for connection properties to set, so it is handled anyway, or if not, then use:
>
> begin
> query1;
> query2;
> query3;
> end;
> to we will have only a single roundtrip for the db.
>
>
> Diffs
> -----
>
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnDbUtil.java bb29410e7d
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java d080df417b
>
>
> Diff: https://reviews.apache.org/r/72392/diff/2/
>
>
> Testing
> -------
>
> Baseline:
> Benchmark (dbProduct) (txnType) Mode Cnt Score Error Units
> TxnHandlerBenchRunner.commitTxn ORACLE DEFAULT ss 100 42.988 ± 4.569 ms/op
> TxnHandlerBenchRunner.commitTxn ORACLE READ_ONLY ss 100 45.029 ± 4.686 ms/op
>
> After patch:
> Benchmark (dbProduct) (txnType) Mode Cnt Score Error Units
> TxnHandlerBenchRunner.commitTxn ORACLE DEFAULT ss 100 36.208 ± 3.869 ms/op
> TxnHandlerBenchRunner.commitTxn ORACLE READ_ONLY ss 100 37.038 ± 3.746 ms/op
>
>
> Thanks,
>
> Peter Vary
>
>
Re: Review Request 72392: HIVE-23103 Oracle statement batching
Posted by Peter Vary via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72392/
-----------------------------------------------------------
(Updated ápr. 21, 2020, 12:41 du)
Review request for hive, Denys Kuzmenko and Marton Bod.
Changes
-------
Addessed Marton's comment
Bugs: HIVE-23103
https://issues.apache.org/jira/browse/HIVE-23103
Repository: hive-git
Description
-------
Examine how to really get better performance for oracle statement batches.
Oracle JDBC doc describes:
The Oracle implementation of standard update batching does not implement true batching for generic statements and callable statements. Even though Oracle JDBC supports the use of standard batching for Statement and CallableStatement objects, you are unlikely to see performance improvement.
I would look for connection properties to set, so it is handled anyway, or if not, then use:
begin
query1;
query2;
query3;
end;
to we will have only a single roundtrip for the db.
Diffs (updated)
-----
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnDbUtil.java bb29410e7d
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java d080df417b
Diff: https://reviews.apache.org/r/72392/diff/2/
Changes: https://reviews.apache.org/r/72392/diff/1-2/
Testing
-------
Baseline:
Benchmark (dbProduct) (txnType) Mode Cnt Score Error Units
TxnHandlerBenchRunner.commitTxn ORACLE DEFAULT ss 100 42.988 ± 4.569 ms/op
TxnHandlerBenchRunner.commitTxn ORACLE READ_ONLY ss 100 45.029 ± 4.686 ms/op
After patch:
Benchmark (dbProduct) (txnType) Mode Cnt Score Error Units
TxnHandlerBenchRunner.commitTxn ORACLE DEFAULT ss 100 36.208 ± 3.869 ms/op
TxnHandlerBenchRunner.commitTxn ORACLE READ_ONLY ss 100 37.038 ± 3.746 ms/op
Thanks,
Peter Vary