You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@orc.apache.org by do...@apache.org on 2021/01/18 17:38:00 UTC
[orc] branch master updated: ORC-736: Upgrade Hive to 3.1.2 (#629)
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/orc.git
The following commit(s) were added to refs/heads/master by this push:
new 51a88c7 ORC-736: Upgrade Hive to 3.1.2 (#629)
51a88c7 is described below
commit 51a88c768b2ec29eb84ce9d3a297b8829255c1b7
Author: Dongjoon Hyun <do...@apache.org>
AuthorDate: Mon Jan 18 09:37:13 2021 -0800
ORC-736: Upgrade Hive to 3.1.2 (#629)
### What changes were proposed in this pull request?
This PR aims to upgrade Hive library from 2.3.3 to 3.1.2 in benchmark module.
### Why are the changes needed?
To depend on the latest one.
### How was this patch tested?
Manual.
```bash
$ cd java/bench
$ mvn package
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for ORC Benchmarks 1.7.0-SNAPSHOT:
[INFO]
[INFO] ORC Benchmarks ..................................... SUCCESS [ 1.066 s]
[INFO] ORC Benchmarks Core ................................ SUCCESS [ 7.244 s]
[INFO] ORC Benchmarks Hive ................................ SUCCESS [ 9.738 s]
[INFO] ORC Benchmarks Spark ............................... SUCCESS [ 21.330 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 39.505 s
[INFO] Finished at: 2021-01-17T20:02:46-08:00
[INFO] ------------------------------------------------------------------------
$ java -jar hive/target/orc-benchmarks-hive-*-uber.jar read-some -i 1 -I 1 ~/data
# JMH version: 1.20
# VM version: JDK 1.8.0_275, VM 25.275-b01
# VM invoker: /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java
# VM options: -server -Xms256m -Xmx2g -Dbench.root.dir=/home/dongjoon/data
# Warmup: 2 iterations, 10 s each
# Measurement: 5 iterations, 10 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Average time, time/op
# Benchmark: org.apache.orc.bench.hive.ColumnProjectionBenchmark.orc
# Parameters: (compression = none, dataset = github)
# Run progress: 0.00% complete, ETA 00:21:00
# Fork: 1 of 1
# Warmup Iteration 1:
Records: 83917136
Invocations: 8
Reads: 12472
Bytes: 5405923304
1387134.776 us/op
...
# Run complete. Total time: 00:06:42
Benchmark (compression) (dataset) Mode Cnt Score Error Units
ColumnProjectionBenchmark.orc none github avgt 1462277.759 us/op
ColumnProjectionBenchmark.orc:bytesPerRecord none github avgt 64.420 #
ColumnProjectionBenchmark.orc:perRecord none github avgt 0.139 us/op
ColumnProjectionBenchmark.orc:reads none github avgt 1559.000 #
ColumnProjectionBenchmark.orc:records none github avgt 10489642.000 #
ColumnProjectionBenchmark.orc none sales avgt 276728.920 us/op
ColumnProjectionBenchmark.orc:bytesPerRecord none sales avgt 4.032 #
ColumnProjectionBenchmark.orc:perRecord none sales avgt 0.011 us/op
ColumnProjectionBenchmark.orc:reads none sales avgt 179.000 #
ColumnProjectionBenchmark.orc:records none sales avgt 25000000.000 #
ColumnProjectionBenchmark.orc none taxi avgt 696710.737 us/op
ColumnProjectionBenchmark.orc:bytesPerRecord none taxi avgt 1.766 #
ColumnProjectionBenchmark.orc:perRecord none taxi avgt 0.031 us/op
ColumnProjectionBenchmark.orc:reads none taxi avgt 77.000 #
ColumnProjectionBenchmark.orc:records none taxi avgt 22773249.000 #
ColumnProjectionBenchmark.orc snappy github avgt 1603492.856 us/op
ColumnProjectionBenchmark.orc:bytesPerRecord snappy github avgt 26.457 #
ColumnProjectionBenchmark.orc:perRecord snappy github avgt 0.153 us/op
ColumnProjectionBenchmark.orc:reads snappy github avgt 1442.000 #
ColumnProjectionBenchmark.orc:records snappy github avgt 10489642.000 #
ColumnProjectionBenchmark.orc snappy sales avgt 277843.598 us/op
ColumnProjectionBenchmark.orc:bytesPerRecord snappy sales avgt 4.020 #
ColumnProjectionBenchmark.orc:perRecord snappy sales avgt 0.011 us/op
ColumnProjectionBenchmark.orc:reads snappy sales avgt 185.000 #
ColumnProjectionBenchmark.orc:records snappy sales avgt 25000000.000 #
ColumnProjectionBenchmark.orc snappy taxi avgt 717087.662 us/op
ColumnProjectionBenchmark.orc:bytesPerRecord snappy taxi avgt 1.260 #
ColumnProjectionBenchmark.orc:perRecord snappy taxi avgt 0.031 us/op
ColumnProjectionBenchmark.orc:reads snappy taxi avgt 52.000 #
ColumnProjectionBenchmark.orc:records snappy taxi avgt 22773249.000 #
ColumnProjectionBenchmark.orc gz github avgt 3552399.105 us/op
ColumnProjectionBenchmark.orc:bytesPerRecord gz github avgt 19.254 #
ColumnProjectionBenchmark.orc:perRecord gz github avgt 0.339 us/op
ColumnProjectionBenchmark.orc:reads gz github avgt 1400.000 #
ColumnProjectionBenchmark.orc:records gz github avgt 10489642.000 #
ColumnProjectionBenchmark.orc gz sales avgt 647495.835 us/op
ColumnProjectionBenchmark.orc:bytesPerRecord gz sales avgt 3.990 #
ColumnProjectionBenchmark.orc:perRecord gz sales avgt 0.026 us/op
ColumnProjectionBenchmark.orc:reads gz sales avgt 167.000 #
ColumnProjectionBenchmark.orc:records gz sales avgt 25000000.000 #
ColumnProjectionBenchmark.orc gz taxi avgt 805176.942 us/op
ColumnProjectionBenchmark.orc:bytesPerRecord gz taxi avgt 0.822 #
ColumnProjectionBenchmark.orc:perRecord gz taxi avgt 0.035 us/op
ColumnProjectionBenchmark.orc:reads gz taxi avgt 40.000 #
ColumnProjectionBenchmark.orc:records gz taxi avgt 22773249.000 #
ColumnProjectionBenchmark.parquet none github avgt 5167469.958 us/op
ColumnProjectionBenchmark.parquet:bytesPerRecord none github avgt 81.186 #
ColumnProjectionBenchmark.parquet:perRecord none github avgt 0.493 us/op
ColumnProjectionBenchmark.parquet:reads none github avgt 7777.000 #
ColumnProjectionBenchmark.parquet:records none github avgt 10489642.000 #
ColumnProjectionBenchmark.parquet none sales avgt 2255171.823 us/op
ColumnProjectionBenchmark.parquet:bytesPerRecord none sales avgt 16.013 #
ColumnProjectionBenchmark.parquet:perRecord none sales avgt 0.090 us/op
ColumnProjectionBenchmark.parquet:reads none sales avgt 113.000 #
ColumnProjectionBenchmark.parquet:records none sales avgt 25000000.000 #
ColumnProjectionBenchmark.parquet none taxi avgt 2071926.094 us/op
ColumnProjectionBenchmark.parquet:bytesPerRecord none taxi avgt 7.448 #
ColumnProjectionBenchmark.parquet:perRecord none taxi avgt 0.091 us/op
ColumnProjectionBenchmark.parquet:reads none taxi avgt 16.000 #
ColumnProjectionBenchmark.parquet:records none taxi avgt 22773249.000 #
ColumnProjectionBenchmark.parquet snappy github avgt 5536923.293 us/op
ColumnProjectionBenchmark.parquet:bytesPerRecord snappy github avgt 29.169 #
ColumnProjectionBenchmark.parquet:perRecord snappy github avgt 0.528 us/op
ColumnProjectionBenchmark.parquet:reads snappy github avgt 7054.000 #
ColumnProjectionBenchmark.parquet:records snappy github avgt 10489642.000 #
ColumnProjectionBenchmark.parquet snappy sales avgt 2746276.721 us/op
ColumnProjectionBenchmark.parquet:bytesPerRecord snappy sales avgt 10.037 #
ColumnProjectionBenchmark.parquet:perRecord snappy sales avgt 0.110 us/op
ColumnProjectionBenchmark.parquet:reads snappy sales avgt 95.000 #
ColumnProjectionBenchmark.parquet:records snappy sales avgt 25000000.000 #
ColumnProjectionBenchmark.parquet snappy taxi avgt 2265655.055 us/op
ColumnProjectionBenchmark.parquet:bytesPerRecord snappy taxi avgt 1.840 #
ColumnProjectionBenchmark.parquet:perRecord snappy taxi avgt 0.099 us/op
ColumnProjectionBenchmark.parquet:reads snappy taxi avgt 12.000 #
ColumnProjectionBenchmark.parquet:records snappy taxi avgt 22773249.000 #
ColumnProjectionBenchmark.parquet gz github avgt 6789831.313 us/op
ColumnProjectionBenchmark.parquet:bytesPerRecord gz github avgt 21.531 #
ColumnProjectionBenchmark.parquet:perRecord gz github avgt 0.647 us/op
ColumnProjectionBenchmark.parquet:reads gz github avgt 6830.000 #
ColumnProjectionBenchmark.parquet:records gz github avgt 10489642.000 #
ColumnProjectionBenchmark.parquet gz sales avgt 3366669.518 us/op
ColumnProjectionBenchmark.parquet:bytesPerRecord gz sales avgt 6.301 #
ColumnProjectionBenchmark.parquet:perRecord gz sales avgt 0.135 us/op
ColumnProjectionBenchmark.parquet:reads gz sales avgt 77.000 #
ColumnProjectionBenchmark.parquet:records gz sales avgt 25000000.000 #
ColumnProjectionBenchmark.parquet gz taxi avgt 2177482.302 us/op
ColumnProjectionBenchmark.parquet:bytesPerRecord gz taxi avgt 0.891 #
ColumnProjectionBenchmark.parquet:perRecord gz taxi avgt 0.096 us/op
ColumnProjectionBenchmark.parquet:reads gz taxi avgt 12.000 #
ColumnProjectionBenchmark.parquet:records gz taxi avgt 22773249.000 #
```
---
java/bench/pom.xml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/java/bench/pom.xml b/java/bench/pom.xml
index 170fb71..d1073bb 100644
--- a/java/bench/pom.xml
+++ b/java/bench/pom.xml
@@ -38,7 +38,7 @@
<avro.version>1.8.2</avro.version>
<hadoop.version>2.7.3</hadoop.version>
- <hive.version>2.3.3</hive.version>
+ <hive.version>3.1.2</hive.version>
<jmh.version>1.20</jmh.version>
<orc.version>${project.version}</orc.version>
<parquet.version>1.8.3</parquet.version>