You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Tim Armstrong (Code Review)" <ge...@cloudera.org> on 2018/06/01 14:08:43 UTC
[Impala-ASF-CR] IMPALA-7078: improve memory consumption of wide Avro scans
Hello Dan Hecht,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/10550
to look at the new patch set (#6).
Change subject: IMPALA-7078: improve memory consumption of wide Avro scans
......................................................................
IMPALA-7078: improve memory consumption of wide Avro scans
Revert to the pre-IMPALA-3905 algorithm for deciding when to return a
batch from an Avro scan. The post-IMPALA-3905 algorithm is bad for
wide tables where there are only a small number of rows per Avro block.
Optimise memory transfer for selective scans - don't attach unused
decompression buffers to the output batch. Combined with the previous
change, this dramatically reduces the amount of memory transferred out
of scanner threads for selective scans of wide tables.
Cap the maximum row batch queue size at 5 * max_num_scanner_threads_
so that we guarantee lower amounts of memory in the row batch queue
when num_scanner_threads is set, rather than just achieving it
statistically because of the producer running slower relative to
consumer. It does not reduce the default significantly on typical
server configurations that would have 24+ cores except under high
concurrency or low memory environments where the number of scanner
threads is limited. We should evaluate reducing the default further
or otherwise better controlling memory consumption in a follow-up,
based on experiments.
Includes some observability improvements including additional
counters that will help diagnose issues like this more easily:
* Add counters to give some insight into row batch queue. Here's
an excerpt:
- RowBatchBytesEnqueued: 20.89 MB (21903380)
- RowBatchQueueCapacity: 5 (5)
- RowBatchQueueGetWaitTime: 59.187ms
- RowBatchQueuePeakMemoryUsage: 8.85 MB (9279347)
- RowBatchQueuePutWaitTime: 0.000ns
- RowBatchesEnqueued: 6 (6)
* Don't create AverageScannerThreadConcurrency for MT scan node where
it's not actually used.
* Track the row batch queue memory consumption against a sub-tracker
HDFS_SCAN_NODE (id=2): Reservation=48.00 MB OtherMemory=588.00 KB Total=48.57 MB Peak=48.62 MB
Queued Batches: Total=588.00 KB Peak=637.00 KB
Ran the repro in the JIRA. Memory consumption was reduced from ~500MB
to ~220MB on my system.
Testing:
* Ran stress test for an hour on uncompressed and 3 hours on snappy-compressed avro.
* Debug exhaustive tests passed.
* ASAN core tests passed.
Perf:
- Parquet TPC-H scale factor 60 on one impalad showed no change in perf
- Avro/Snappy scale factor 20 showed a small improvement:
+----------+---------------------+---------+------------+------------+----------------+
| Workload | File Format | Avg (s) | Delta(Avg) | GeoMean(s) | Delta(GeoMean) |
+----------+---------------------+---------+------------+------------+----------------+
| TPCH(20) | avro / snap / block | 9.86 | -2.23% | 7.83 | -2.37% |
+----------+---------------------+---------+------------+------------+----------------+
+----------+----------+----------------------+--------+-------------+------------+-----------+----------------+-------------+-------+
| Workload | Query | File Format | Avg(s) | Base Avg(s) | Delta(Avg) | StdDev(%) | Base StdDev(%) | Num Clients | Iters |
+----------+----------+----------------------+--------+-------------+------------+-----------+----------------+-------------+-------+
| TPCH(20) | TPCH-Q6 | avro / block / block | 5.59 | 5.17 | +8.10% | 0.75% | 0.71% | 1 | 30 |
| TPCH(20) | TPCH-Q14 | avro / block / block | 6.31 | 5.89 | +7.21% | 0.68% | 0.74% | 1 | 30 |
| TPCH(20) | TPCH-Q15 | avro / block / block | 11.32 | 10.64 | +6.41% | 0.37% | 0.46% | 1 | 30 |
| TPCH(20) | TPCH-Q12 | avro / block / block | 8.57 | 8.14 | +5.23% | 0.67% | 0.81% | 1 | 30 |
| TPCH(20) | TPCH-Q13 | avro / block / block | 6.72 | 6.54 | +2.72% | 0.77% | 0.73% | 1 | 30 |
| TPCH(20) | TPCH-Q4 | avro / block / block | 11.76 | 11.61 | +1.32% | 0.60% | 0.61% | 1 | 30 |
| TPCH(20) | TPCH-Q7 | avro / block / block | 14.43 | 14.26 | +1.21% | 1.14% | 0.35% | 1 | 30 |
| TPCH(20) | TPCH-Q21 | avro / block / block | 34.12 | 34.25 | -0.36% | 0.27% | 0.24% | 1 | 30 |
| TPCH(20) | TPCH-Q20 | avro / block / block | 8.49 | 8.52 | -0.38% | 0.45% | 0.54% | 1 | 30 |
| TPCH(20) | TPCH-Q1 | avro / block / block | 6.99 | 7.02 | -0.38% | 0.96% | 0.65% | 1 | 30 |
| TPCH(20) | TPCH-Q22 | avro / block / block | 2.44 | 2.47 | -1.09% | 1.81% | 1.47% | 1 | 30 |
| TPCH(20) | TPCH-Q11 | avro / block / block | 1.99 | 2.02 | -1.57% | 1.95% | 1.90% | 1 | 30 |
| TPCH(20) | TPCH-Q17 | avro / block / block | 13.57 | 13.79 | -1.63% | 1.53% | 1.31% | 1 | 30 |
| TPCH(20) | TPCH-Q18 | avro / block / block | 21.93 | 22.31 | -1.72% | 0.31% | 0.34% | 1 | 30 |
| TPCH(20) | TPCH-Q8 | avro / block / block | 9.05 | 9.31 | -2.81% | 0.85% | 0.72% | 1 | 30 |
| TPCH(20) | TPCH-Q19 | avro / block / block | 7.20 | 7.41 | -2.91% | 0.72% | 0.52% | 1 | 30 |
| TPCH(20) | TPCH-Q9 | avro / block / block | 14.25 | 14.73 | -3.29% | 0.45% | 0.33% | 1 | 30 |
| TPCH(20) | TPCH-Q2 | avro / block / block | 2.69 | 2.88 | -6.66% | 1.17% | 1.52% | 1 | 30 |
| TPCH(20) | TPCH-Q16 | avro / block / block | 2.12 | 2.30 | -7.82% | 2.56% | 2.10% | 1 | 30 |
| TPCH(20) | TPCH-Q3 | avro / block / block | 9.68 | 11.24 | -13.85% | 0.46% | 0.50% | 1 | 30 |
| TPCH(20) | TPCH-Q10 | avro / block / block | 8.92 | 10.66 | -16.33% | 0.75% | 0.49% | 1 | 30 |
| TPCH(20) | TPCH-Q5 | avro / block / block | 8.76 | 10.69 | -18.08% | 0.64% | 0.49% | 1 | 30 |
+----------+----------+----------------------+--------+-------------+------------+-----------+----------------+-------------+-------+
Change-Id: Iebd2600b4784fd19696c9b92eefb7d7e9db0c80b
---
M be/src/exec/hdfs-avro-scanner.cc
M be/src/exec/hdfs-scan-node-base.cc
M be/src/exec/hdfs-scan-node.cc
M be/src/exec/hdfs-scan-node.h
M be/src/exec/hdfs-scanner.cc
M be/src/exec/scan-node.h
M be/src/runtime/mem-pool.cc
M be/src/runtime/mem-pool.h
M be/src/runtime/mem-tracker-test.cc
M be/src/runtime/mem-tracker.cc
M be/src/runtime/mem-tracker.h
M be/src/runtime/row-batch.cc
M be/src/runtime/row-batch.h
13 files changed, 248 insertions(+), 76 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/10550/6
--
To view, visit http://gerrit.cloudera.org:8080/10550
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iebd2600b4784fd19696c9b92eefb7d7e9db0c80b
Gerrit-Change-Number: 10550
Gerrit-PatchSet: 6
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>