You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Tim Armstrong (Code Review)" <ge...@cloudera.org> on 2019/11/18 17:11:12 UTC
[Impala-ASF-CR] IMPALA-9134: pipeline data stream sender flush
Tim Armstrong has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/14717 )
Change subject: IMPALA-9134: pipeline data stream sender flush
......................................................................
IMPALA-9134: pipeline data stream sender flush
This changes the logic so that the last TransmitData
and EndDataStream RPCs can be sent to receivers in
parallel, instead of them being serialized in FlushAndSendEos().
This was a problem because each call to FlushAndSendEos() would
wait for one or two network round-trips, so previously the flush
could easily take sum(2 * RTT). After this change the sending
and waiting for RPCs is pipeline so the time taken should be
closer to 2 * RTT.
Perf:
No measurable change on local TPC-H (where network latency isn't a
significant factor).
+----------+-----------------------+---------+------------+------------+----------------+
| Workload | File Format | Avg (s) | Delta(Avg) | GeoMean(s) | Delta(GeoMean) |
+----------+-----------------------+---------+------------+------------+----------------+
| TPCH(30) | parquet / none / none | 7.64 | -1.37% | 5.00 | +0.29% |
+----------+-----------------------+---------+------------+------------+----------------+
+----------+----------+-----------------------+--------+-------------+------------+------------+----------------+-------+----------------+---------+-------+
| Workload | Query | File Format | Avg(s) | Base Avg(s) | Delta(Avg) | StdDev(%) | Base StdDev(%) | Iters | Median Diff(%) | MW Zval | Tval |
+----------+----------+-----------------------+--------+-------------+------------+------------+----------------+-------+----------------+---------+-------+
| TPCH(30) | TPCH-Q14 | parquet / none / none | 3.75 | 3.39 | +10.48% | * 19.43% * | 8.88% | 7 | +7.26% | 0.48 | 1.19 |
| TPCH(30) | TPCH-Q13 | parquet / none / none | 13.55 | 12.78 | +6.05% | 5.43% | 4.75% | 7 | +5.09% | 1.76 | 2.14 |
| TPCH(30) | TPCH-Q2 | parquet / none / none | 1.30 | 1.26 | +3.63% | 5.53% | 3.27% | 7 | +3.93% | 1.28 | 1.46 |
| TPCH(30) | TPCH-Q12 | parquet / none / none | 2.65 | 2.56 | +3.44% | 4.23% | 2.92% | 7 | +3.68% | 1.28 | 1.73 |
| TPCH(30) | TPCH-Q15 | parquet / none / none | 3.80 | 3.76 | +0.91% | 0.81% | 1.80% | 7 | +1.42% | 1.12 | 1.22 |
| TPCH(30) | TPCH-Q8 | parquet / none / none | 5.23 | 5.13 | +2.11% | 5.41% | 0.48% | 7 | +0.04% | 0.16 | 1.01 |
| TPCH(30) | TPCH-Q10 | parquet / none / none | 5.77 | 5.62 | +2.70% | 9.84% | 2.79% | 7 | -0.80% | -0.96 | 0.68 |
| TPCH(30) | TPCH-Q21 | parquet / none / none | 28.82 | 28.62 | +0.70% | 0.59% | 0.37% | 7 | +0.68% | 1.92 | 2.67 |
| TPCH(30) | TPCH-Q6 | parquet / none / none | 1.32 | 1.31 | +0.82% | 0.36% | 1.53% | 7 | +0.23% | 1.28 | 1.38 |
| TPCH(30) | TPCH-Q11 | parquet / none / none | 1.30 | 1.29 | +0.60% | 5.11% | 2.94% | 7 | +0.14% | 0.16 | 0.27 |
| TPCH(30) | TPCH-Q19 | parquet / none / none | 4.52 | 4.49 | +0.57% | 2.98% | 1.58% | 7 | +0.12% | 0.32 | 0.44 |
| TPCH(30) | TPCH-Q16 | parquet / none / none | 2.78 | 2.77 | +0.35% | 3.21% | 2.02% | 7 | +0.17% | 0.64 | 0.25 |
| TPCH(30) | TPCH-Q3 | parquet / none / none | 5.65 | 5.65 | -0.04% | 1.62% | 1.91% | 7 | -0.06% | -0.32 | -0.04 |
| TPCH(30) | TPCH-Q1 | parquet / none / none | 10.88 | 10.89 | -0.11% | 0.76% | 0.90% | 7 | +0.01% | 0.00 | -0.26 |
| TPCH(30) | TPCH-Q4 | parquet / none / none | 2.18 | 2.18 | -0.38% | 1.33% | 2.48% | 7 | -0.04% | -0.16 | -0.36 |
| TPCH(30) | TPCH-Q5 | parquet / none / none | 3.94 | 3.95 | -0.27% | 2.37% | 1.33% | 7 | -0.26% | -1.12 | -0.26 |
| TPCH(30) | TPCH-Q7 | parquet / none / none | 22.37 | 22.41 | -0.19% | 2.94% | 1.92% | 7 | -0.66% | -0.80 | -0.15 |
| TPCH(30) | TPCH-Q22 | parquet / none / none | 3.68 | 3.72 | -1.14% | 4.95% | 3.99% | 7 | +0.13% | 0.00 | -0.48 |
| TPCH(30) | TPCH-Q20 | parquet / none / none | 2.83 | 2.86 | -1.21% | 2.69% | 2.20% | 7 | -1.60% | -0.64 | -0.93 |
| TPCH(30) | TPCH-Q17 | parquet / none / none | 4.92 | 5.04 | -2.46% | 3.11% | 2.63% | 7 | -2.00% | -1.44 | -1.62 |
| TPCH(30) | TPCH-Q9 | parquet / none / none | 16.32 | 17.19 | -5.03% | 2.76% | 4.56% | 7 | -4.00% | -2.24 | -2.53 |
| TPCH(30) | TPCH-Q18 | parquet / none / none | 20.45 | 23.45 | -12.82% | 5.49% | * 23.11% * | 7 | -4.89% | -0.96 | -1.44 |
+----------+----------+-----------------------+--------+-------------+------------+------------+----------------+-------+----------------+---------+-------+
Change-Id: If66add99e4370f8faf22761b336205cc9f9f1867
---
M be/src/runtime/krpc-data-stream-sender.cc
1 file changed, 63 insertions(+), 30 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/17/14717/2
--
To view, visit http://gerrit.cloudera.org:8080/14717
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If66add99e4370f8faf22761b336205cc9f9f1867
Gerrit-Change-Number: 14717
Gerrit-PatchSet: 2
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>