You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by ta...@apache.org on 2018/06/07 17:12:26 UTC

[03/10] impala git commit: IMPALA-7078: Part 1: improve memory consumption of wide Avro scans

IMPALA-7078: Part 1: improve memory consumption of wide Avro scans

Revert to the pre-IMPALA-3905 algorithm for deciding when to return a
batch from an Avro scan. The post-IMPALA-3905 algorithm is bad for
wide tables where there are only a small number of rows per Avro block.

Optimise memory transfer for selective scans - don't attach unused
decompression buffers to the output batch. Combined with the previous
change, this dramatically reduces the amount of memory transferred out
of scanner threads for selective scans of wide tables.

Includes some observability improvements including additional
counters that will help diagnose issues like this more easily:
* Add counters to give some insight into row batch queue. Here's
  an excerpt:
   - RowBatchBytesEnqueued: 20.89 MB (21903380)
   - RowBatchQueueCapacity: 5 (5)
   - RowBatchQueueGetWaitTime: 59.187ms
   - RowBatchQueuePeakMemoryUsage: 8.85 MB (9279347)
   - RowBatchQueuePutWaitTime: 0.000ns
   - RowBatchesEnqueued: 6 (6)
* Don't create AverageScannerThreadConcurrency for MT scan node where
  it's not actually used.
* Track the row batch queue memory consumption against a sub-tracker
  HDFS_SCAN_NODE (id=2): Reservation=48.00 MB OtherMemory=588.00 KB Total=48.57 MB Peak=48.62 MB
    Queued Batches: Total=588.00 KB Peak=637.00 KB

Ran the repro in the JIRA. Memory consumption was reduced from ~500MB
to ~220MB on my system.

Testing:
* Ran stress test for an hour on uncompressed and 3 hours on snappy-compressed avro.
* Debug exhaustive tests passed.
* ASAN core tests passed.

Perf:
- Parquet TPC-H scale factor 60 on one impalad showed no change in perf
- Avro/Snappy scale factor 20 showed a small improvement:
+----------+---------------------+---------+------------+------------+----------------+
| Workload | File Format         | Avg (s) | Delta(Avg) | GeoMean(s) | Delta(GeoMean) |
+----------+---------------------+---------+------------+------------+----------------+
| TPCH(20) | avro / snap / block | 9.86    | -2.23%     | 7.83       | -2.37%         |
+----------+---------------------+---------+------------+------------+----------------+

+----------+----------+----------------------+--------+-------------+------------+-----------+----------------+-------------+-------+
| Workload | Query    | File Format          | Avg(s) | Base Avg(s) | Delta(Avg) | StdDev(%) | Base StdDev(%) | Num Clients | Iters |
+----------+----------+----------------------+--------+-------------+------------+-----------+----------------+-------------+-------+
| TPCH(20) | TPCH-Q6  | avro / block / block | 5.59   | 5.17        |   +8.10%   |   0.75%   |   0.71%        | 1           | 30    |
| TPCH(20) | TPCH-Q14 | avro / block / block | 6.31   | 5.89        |   +7.21%   |   0.68%   |   0.74%        | 1           | 30    |
| TPCH(20) | TPCH-Q15 | avro / block / block | 11.32  | 10.64       |   +6.41%   |   0.37%   |   0.46%        | 1           | 30    |
| TPCH(20) | TPCH-Q12 | avro / block / block | 8.57   | 8.14        |   +5.23%   |   0.67%   |   0.81%        | 1           | 30    |
| TPCH(20) | TPCH-Q13 | avro / block / block | 6.72   | 6.54        |   +2.72%   |   0.77%   |   0.73%        | 1           | 30    |
| TPCH(20) | TPCH-Q4  | avro / block / block | 11.76  | 11.61       |   +1.32%   |   0.60%   |   0.61%        | 1           | 30    |
| TPCH(20) | TPCH-Q7  | avro / block / block | 14.43  | 14.26       |   +1.21%   |   1.14%   |   0.35%        | 1           | 30    |
| TPCH(20) | TPCH-Q21 | avro / block / block | 34.12  | 34.25       |   -0.36%   |   0.27%   |   0.24%        | 1           | 30    |
| TPCH(20) | TPCH-Q20 | avro / block / block | 8.49   | 8.52        |   -0.38%   |   0.45%   |   0.54%        | 1           | 30    |
| TPCH(20) | TPCH-Q1  | avro / block / block | 6.99   | 7.02        |   -0.38%   |   0.96%   |   0.65%        | 1           | 30    |
| TPCH(20) | TPCH-Q22 | avro / block / block | 2.44   | 2.47        |   -1.09%   |   1.81%   |   1.47%        | 1           | 30    |
| TPCH(20) | TPCH-Q11 | avro / block / block | 1.99   | 2.02        |   -1.57%   |   1.95%   |   1.90%        | 1           | 30    |
| TPCH(20) | TPCH-Q17 | avro / block / block | 13.57  | 13.79       |   -1.63%   |   1.53%   |   1.31%        | 1           | 30    |
| TPCH(20) | TPCH-Q18 | avro / block / block | 21.93  | 22.31       |   -1.72%   |   0.31%   |   0.34%        | 1           | 30    |
| TPCH(20) | TPCH-Q8  | avro / block / block | 9.05   | 9.31        |   -2.81%   |   0.85%   |   0.72%        | 1           | 30    |
| TPCH(20) | TPCH-Q19 | avro / block / block | 7.20   | 7.41        |   -2.91%   |   0.72%   |   0.52%        | 1           | 30    |
| TPCH(20) | TPCH-Q9  | avro / block / block | 14.25  | 14.73       |   -3.29%   |   0.45%   |   0.33%        | 1           | 30    |
| TPCH(20) | TPCH-Q2  | avro / block / block | 2.69   | 2.88        |   -6.66%   |   1.17%   |   1.52%        | 1           | 30    |
| TPCH(20) | TPCH-Q16 | avro / block / block | 2.12   | 2.30        |   -7.82%   |   2.56%   |   2.10%        | 1           | 30    |
| TPCH(20) | TPCH-Q3  | avro / block / block | 9.68   | 11.24       |   -13.85%  |   0.46%   |   0.50%        | 1           | 30    |
| TPCH(20) | TPCH-Q10 | avro / block / block | 8.92   | 10.66       |   -16.33%  |   0.75%   |   0.49%        | 1           | 30    |
| TPCH(20) | TPCH-Q5  | avro / block / block | 8.76   | 10.69       |   -18.08%  |   0.64%   |   0.49%        | 1           | 30    |
+----------+----------+----------------------+--------+-------------+------------+-----------+----------------+-------------+-------+

Change-Id: Iebd2600b4784fd19696c9b92eefb7d7e9db0c80b
Reviewed-on: http://gerrit.cloudera.org:8080/10550
Reviewed-by: Tim Armstrong <ta...@cloudera.com>
Tested-by: Tim Armstrong <ta...@cloudera.com>


Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/3c376950
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/3c376950
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/3c376950

Branch: refs/heads/2.x
Commit: 3c376950902fdab628d2855872555472ec4668f9
Parents: 5c1dd4f
Author: Tim Armstrong <ta...@cloudera.com>
Authored: Wed May 30 11:47:52 2018 -0700
Committer: Impala Public Jenkins <im...@gerrit.cloudera.org>
Committed: Thu Jun 7 11:10:16 2018 +0000

----------------------------------------------------------------------
 be/src/exec/hdfs-avro-scanner.cc   | 140 ++++++++++++++++++--------------
 be/src/exec/hdfs-scan-node-base.cc |   2 -
 be/src/exec/hdfs-scan-node.cc      |  23 ++++++
 be/src/exec/hdfs-scan-node.h       |  15 ++++
 be/src/exec/hdfs-scanner.cc        |   8 +-
 be/src/exec/scan-node.h            |   4 +-
 be/src/runtime/mem-pool.cc         |  14 ++--
 be/src/runtime/mem-pool.h          |   3 +
 be/src/runtime/mem-tracker-test.cc |  61 ++++++++++++++
 be/src/runtime/mem-tracker.cc      |  19 +++++
 be/src/runtime/mem-tracker.h       |   6 ++
 be/src/runtime/row-batch.cc        |   6 ++
 be/src/runtime/row-batch.h         |   3 +
 13 files changed, 233 insertions(+), 71 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/impala/blob/3c376950/be/src/exec/hdfs-avro-scanner.cc
----------------------------------------------------------------------
diff --git a/be/src/exec/hdfs-avro-scanner.cc b/be/src/exec/hdfs-avro-scanner.cc
index e74b589..9a92763 100644
--- a/be/src/exec/hdfs-avro-scanner.cc
+++ b/be/src/exec/hdfs-avro-scanner.cc
@@ -493,73 +493,93 @@ Status HdfsAvroScanner::InitNewRange() {
 }
 
 Status HdfsAvroScanner::ProcessRange(RowBatch* row_batch) {
-  if (record_pos_ == num_records_in_block_) {
-    // Read new data block
-    RETURN_IF_FALSE(stream_->ReadZLong(&num_records_in_block_, &parse_status_));
-    if (num_records_in_block_ < 0) {
-      return Status(TErrorCode::AVRO_INVALID_RECORD_COUNT, stream_->filename(),
-          num_records_in_block_, stream_->file_offset());
-    }
-    int64_t compressed_size;
-    RETURN_IF_FALSE(stream_->ReadZLong(&compressed_size, &parse_status_));
-    if (compressed_size < 0) {
-      return Status(TErrorCode::AVRO_INVALID_COMPRESSED_SIZE, stream_->filename(),
-          compressed_size, stream_->file_offset());
-    }
-    uint8_t* compressed_data;
-    RETURN_IF_FALSE(stream_->ReadBytes(
-        compressed_size, &compressed_data, &parse_status_));
-
-    if (header_->is_compressed) {
-      if (header_->compression_type == THdfsCompression::SNAPPY) {
-        // Snappy-compressed data block includes trailing 4-byte checksum,
-        // decompressor_ doesn't expect this
-        compressed_size -= SnappyDecompressor::TRAILING_CHECKSUM_LEN;
+  // Process blocks until we hit eos, the limit or the batch fills up. Check
+  // AtCapacity() at the end of the loop to guarantee that we process at least one row
+  // so that we make progress even if the batch starts off with AtCapacity() == true,
+  // which can happen if the tuple buffer is > 8MB.
+  DCHECK_GT(row_batch->capacity(), row_batch->num_rows());
+  while (!eos_ && !scan_node_->ReachedLimit()) {
+    if (record_pos_ == num_records_in_block_) {
+      // Read new data block
+      RETURN_IF_FALSE(stream_->ReadZLong(&num_records_in_block_, &parse_status_));
+      if (num_records_in_block_ < 0) {
+        return Status(TErrorCode::AVRO_INVALID_RECORD_COUNT, stream_->filename(),
+            num_records_in_block_, stream_->file_offset());
       }
-      SCOPED_TIMER(decompress_timer_);
-      RETURN_IF_ERROR(decompressor_->ProcessBlock(false, compressed_size,
-          compressed_data, &data_block_len_, &data_block_));
-    } else {
-      data_block_ = compressed_data;
-      data_block_len_ = compressed_size;
+      int64_t compressed_size;
+      RETURN_IF_FALSE(stream_->ReadZLong(&compressed_size, &parse_status_));
+      if (compressed_size < 0) {
+        return Status(TErrorCode::AVRO_INVALID_COMPRESSED_SIZE, stream_->filename(),
+            compressed_size, stream_->file_offset());
+      }
+      uint8_t* compressed_data;
+      RETURN_IF_FALSE(stream_->ReadBytes(
+          compressed_size, &compressed_data, &parse_status_));
+
+      if (header_->is_compressed) {
+        if (header_->compression_type == THdfsCompression::SNAPPY) {
+          // Snappy-compressed data block includes trailing 4-byte checksum,
+          // decompressor_ doesn't expect this
+          compressed_size -= SnappyDecompressor::TRAILING_CHECKSUM_LEN;
+        }
+        SCOPED_TIMER(decompress_timer_);
+        RETURN_IF_ERROR(decompressor_->ProcessBlock(false, compressed_size,
+            compressed_data, &data_block_len_, &data_block_));
+      } else {
+        data_block_ = compressed_data;
+        data_block_len_ = compressed_size;
+      }
+      data_block_end_ = data_block_ + data_block_len_;
+      record_pos_ = 0;
     }
-    data_block_end_ = data_block_ + data_block_len_;
-    record_pos_ = 0;
-  }
 
-  // Process block data
-  while (record_pos_ != num_records_in_block_) {
-    SCOPED_TIMER(scan_node_->materialize_tuple_timer());
-
-    Tuple* tuple = tuple_;
-    TupleRow* tuple_row = row_batch->GetRow(row_batch->AddRow());
-    int max_tuples = row_batch->capacity() - row_batch->num_rows();
-    max_tuples = min<int64_t>(max_tuples, num_records_in_block_ - record_pos_);
-    int num_to_commit;
-    if (scan_node_->materialized_slots().empty()) {
-      // No slots to materialize (e.g. count(*)), no need to decode data
-      num_to_commit = WriteTemplateTuples(tuple_row, max_tuples);
-    } else if (codegend_decode_avro_data_ != nullptr) {
-      num_to_commit = codegend_decode_avro_data_(this, max_tuples,
-          row_batch->tuple_data_pool(), &data_block_, data_block_end_, tuple, tuple_row);
-    } else {
-      num_to_commit = DecodeAvroData(max_tuples, row_batch->tuple_data_pool(),
-          &data_block_, data_block_end_, tuple, tuple_row);
+    int64_t prev_record_pos = record_pos_;
+    int block_start_row = row_batch->num_rows();
+    // Process the remaining data in the current block. Always process at least one row
+    // to ensure we make progress even if the batch starts off with AtCapacity() == true.
+    DCHECK_GT(row_batch->capacity(), row_batch->num_rows());
+    while (record_pos_ != num_records_in_block_) {
+      SCOPED_TIMER(scan_node_->materialize_tuple_timer());
+
+      Tuple* tuple = tuple_;
+      TupleRow* tuple_row = row_batch->GetRow(row_batch->AddRow());
+      int max_tuples = row_batch->capacity() - row_batch->num_rows();
+      max_tuples = min<int64_t>(max_tuples, num_records_in_block_ - record_pos_);
+      int num_to_commit;
+      if (scan_node_->materialized_slots().empty()) {
+        // No slots to materialize (e.g. count(*)), no need to decode data
+        num_to_commit = WriteTemplateTuples(tuple_row, max_tuples);
+      } else if (codegend_decode_avro_data_ != nullptr) {
+        num_to_commit = codegend_decode_avro_data_(this, max_tuples,
+            row_batch->tuple_data_pool(), &data_block_, data_block_end_, tuple, tuple_row);
+      } else {
+        num_to_commit = DecodeAvroData(max_tuples, row_batch->tuple_data_pool(),
+            &data_block_, data_block_end_, tuple, tuple_row);
+      }
+      RETURN_IF_ERROR(parse_status_);
+      RETURN_IF_ERROR(CommitRows(num_to_commit, row_batch));
+      record_pos_ += max_tuples;
+      COUNTER_ADD(scan_node_->rows_read_counter(), max_tuples);
+      if (row_batch->AtCapacity() || scan_node_->ReachedLimit()) break;
     }
-    RETURN_IF_ERROR(parse_status_);
-    RETURN_IF_ERROR(CommitRows(num_to_commit, row_batch));
-    record_pos_ += max_tuples;
-    COUNTER_ADD(scan_node_->rows_read_counter(), max_tuples);
-    if (row_batch->AtCapacity() || scan_node_->ReachedLimit()) break;
-  }
 
-  if (record_pos_ == num_records_in_block_) {
-    if (decompressor_.get() != nullptr && !decompressor_->reuse_output_buffer()) {
-      row_batch->tuple_data_pool()->AcquireData(data_buffer_pool_.get(), false);
+    if (record_pos_ == num_records_in_block_) {
+      if (decompressor_.get() != nullptr && !decompressor_->reuse_output_buffer()) {
+        if (prev_record_pos == 0 && row_batch->num_rows() == block_start_row) {
+          // Did not return any rows from current block in this or a previous
+          // ProcessRange() call - we can recycle the memory. This optimisation depends on
+          // passing keep_current_chunk = false to the AcquireData() call below, so that
+          // the current chunk only contains data for the current Avro block.
+          data_buffer_pool_->Clear();
+        } else {
+          // Returned rows may reference data buffers - need to attach to batch.
+          row_batch->tuple_data_pool()->AcquireData(data_buffer_pool_.get(), false);
+        }
+      }
+      RETURN_IF_ERROR(ReadSync());
     }
-    RETURN_IF_ERROR(ReadSync());
+    if (row_batch->AtCapacity()) break;
   }
-
   return Status::OK();
 }
 

http://git-wip-us.apache.org/repos/asf/impala/blob/3c376950/be/src/exec/hdfs-scan-node-base.cc
----------------------------------------------------------------------
diff --git a/be/src/exec/hdfs-scan-node-base.cc b/be/src/exec/hdfs-scan-node-base.cc
index 00f3af1..25be748 100644
--- a/be/src/exec/hdfs-scan-node-base.cc
+++ b/be/src/exec/hdfs-scan-node-base.cc
@@ -372,8 +372,6 @@ Status HdfsScanNodeBase::Open(RuntimeState* state) {
   reader_context_->set_active_read_thread_counter(&active_hdfs_read_thread_counter_);
   reader_context_->set_disks_accessed_bitmap(&disks_accessed_bitmap_);
 
-  average_scanner_thread_concurrency_ = runtime_profile()->AddSamplingCounter(
-      AVERAGE_SCANNER_THREAD_CONCURRENCY, &active_scanner_thread_counter_);
   average_hdfs_read_thread_concurrency_ = runtime_profile()->AddSamplingCounter(
       AVERAGE_HDFS_READ_THREAD_CONCURRENCY, &active_hdfs_read_thread_counter_);
 

http://git-wip-us.apache.org/repos/asf/impala/blob/3c376950/be/src/exec/hdfs-scan-node.cc
----------------------------------------------------------------------
diff --git a/be/src/exec/hdfs-scan-node.cc b/be/src/exec/hdfs-scan-node.cc
index e434731..f1d58f2 100644
--- a/be/src/exec/hdfs-scan-node.cc
+++ b/be/src/exec/hdfs-scan-node.cc
@@ -84,6 +84,7 @@ Status HdfsScanNode::GetNext(RuntimeState* state, RowBatch* row_batch, bool* eos
     StopAndFinalizeCounters();
     row_batches_put_timer_->Set(materialized_row_batches_->total_put_wait_time());
     row_batches_get_timer_->Set(materialized_row_batches_->total_get_wait_time());
+    row_batches_peak_mem_consumption_->Set(row_batches_mem_tracker_->peak_consumption());
   }
   return status;
 }
@@ -154,8 +155,19 @@ Status HdfsScanNode::Init(const TPlanNode& tnode, RuntimeState* state) {
 Status HdfsScanNode::Prepare(RuntimeState* state) {
   SCOPED_TIMER(runtime_profile_->total_time_counter());
   RETURN_IF_ERROR(HdfsScanNodeBase::Prepare(state));
+  row_batches_mem_tracker_ = state->obj_pool()->Add(new MemTracker(
+        -1, "Queued Batches", mem_tracker(), false));
+
+  row_batches_enqueued_ =
+      ADD_COUNTER(runtime_profile(), "RowBatchesEnqueued", TUnit::UNIT);
+  row_batch_bytes_enqueued_ =
+      ADD_COUNTER(runtime_profile(), "RowBatchBytesEnqueued", TUnit::BYTES);
   row_batches_get_timer_ = ADD_TIMER(runtime_profile(), "RowBatchQueueGetWaitTime");
   row_batches_put_timer_ = ADD_TIMER(runtime_profile(), "RowBatchQueuePutWaitTime");
+  row_batches_max_capacity_ = runtime_profile()->AddHighWaterMarkCounter(
+      "RowBatchQueueCapacity", TUnit::UNIT);
+  row_batches_peak_mem_consumption_ =
+      ADD_COUNTER(runtime_profile(), "RowBatchQueuePeakMemoryUsage", TUnit::BYTES);
   scanner_thread_reservations_denied_counter_ =
       ADD_COUNTER(runtime_profile(), "NumScannerThreadReservationsDenied", TUnit::UNIT);
   return Status::OK();
@@ -170,10 +182,14 @@ Status HdfsScanNode::Open(RuntimeState* state) {
 
   if (file_descs_.empty() || progress_.done()) return Status::OK();
 
+  average_scanner_thread_concurrency_ = runtime_profile()->AddSamplingCounter(
+      AVERAGE_SCANNER_THREAD_CONCURRENCY, &active_scanner_thread_counter_);
+
   if (runtime_state_->query_options().num_scanner_threads > 0) {
     max_num_scanner_threads_ = runtime_state_->query_options().num_scanner_threads;
   }
   DCHECK_GT(max_num_scanner_threads_, 0);
+
   thread_avail_cb_id_ = runtime_state_->resource_pool()->AddThreadAvailableCb(
       bind<void>(mem_fn(&HdfsScanNode::ThreadTokenAvailableCb), this, _1));
   return Status::OK();
@@ -187,6 +203,7 @@ void HdfsScanNode::Close(RuntimeState* state) {
   }
   scanner_threads_.JoinAll();
   materialized_row_batches_->Cleanup();
+  if (row_batches_mem_tracker_ != nullptr) row_batches_mem_tracker_->Close();
   HdfsScanNodeBase::Close(state);
 }
 
@@ -203,6 +220,12 @@ void HdfsScanNode::TransferToScanNodePool(MemPool* pool) {
 
 void HdfsScanNode::AddMaterializedRowBatch(unique_ptr<RowBatch> row_batch) {
   InitNullCollectionValues(row_batch.get());
+  COUNTER_ADD(row_batches_enqueued_, 1);
+  // Only need to count tuple_data_pool() bytes since after IMPALA-5307, no buffers are
+  // returned from the scan node.
+  COUNTER_ADD(row_batch_bytes_enqueued_,
+      row_batch->tuple_data_pool()->total_reserved_bytes());
+  row_batch->SetMemTracker(row_batches_mem_tracker_);
   materialized_row_batches_->AddBatch(move(row_batch));
 }
 

http://git-wip-us.apache.org/repos/asf/impala/blob/3c376950/be/src/exec/hdfs-scan-node.h
----------------------------------------------------------------------
diff --git a/be/src/exec/hdfs-scan-node.h b/be/src/exec/hdfs-scan-node.h
index d35a40c..a7026df 100644
--- a/be/src/exec/hdfs-scan-node.h
+++ b/be/src/exec/hdfs-scan-node.h
@@ -151,16 +151,31 @@ class HdfsScanNode : public HdfsScanNodeBase {
   /// the number of cores.
   int max_num_scanner_threads_;
 
+  // MemTracker for queued row batches. Initialized in Prepare(). Owned by RuntimeState.
+  MemTracker* row_batches_mem_tracker_ = nullptr;
+
   // Number of times scanner threads were not created because of reservation increase
   // being denied.
   RuntimeProfile::Counter* scanner_thread_reservations_denied_counter_ = nullptr;
 
+  /// The number of row batches enqueued into the row batch queue.
+  RuntimeProfile::Counter* row_batches_enqueued_ = nullptr;
+
+  /// The total bytes of row batches enqueued into the row batch queue.
+  RuntimeProfile::Counter* row_batch_bytes_enqueued_ = nullptr;
+
   /// The wait time for fetching a row batch from the row batch queue.
   RuntimeProfile::Counter* row_batches_get_timer_ = nullptr;
 
   /// The wait time for enqueuing a row batch into the row batch queue.
   RuntimeProfile::Counter* row_batches_put_timer_ = nullptr;
 
+  /// Maximum capacity of the row batch queue.
+  RuntimeProfile::HighWaterMarkCounter* row_batches_max_capacity_ = nullptr;
+
+  /// Peak memory consumption of the materialized batch queue. Updated in Close().
+  RuntimeProfile::Counter* row_batches_peak_mem_consumption_ = nullptr;
+
   /// Tries to spin up as many scanner threads as the quota allows. Called explicitly
   /// (e.g., when adding new ranges) or when threads are available for this scan node.
   void ThreadTokenAvailableCb(ThreadResourcePool* pool);

http://git-wip-us.apache.org/repos/asf/impala/blob/3c376950/be/src/exec/hdfs-scanner.cc
----------------------------------------------------------------------
diff --git a/be/src/exec/hdfs-scanner.cc b/be/src/exec/hdfs-scanner.cc
index 2fd488f..da44e9a 100644
--- a/be/src/exec/hdfs-scanner.cc
+++ b/be/src/exec/hdfs-scanner.cc
@@ -119,6 +119,7 @@ Status HdfsScanner::Open(ScannerContext* context) {
 Status HdfsScanner::ProcessSplit() {
   DCHECK(scan_node_->HasRowBatchQueue());
   HdfsScanNode* scan_node = static_cast<HdfsScanNode*>(scan_node_);
+  bool returned_rows = false;
   do {
     // IMPALA-3798, IMPALA-3804: For sequence-based files, the filters are only
     // applied in HdfsScanNode::ProcessSplit()
@@ -132,9 +133,10 @@ Status HdfsScanner::ProcessSplit() {
     unique_ptr<RowBatch> batch = std::make_unique<RowBatch>(scan_node_->row_desc(),
         state_->batch_size(), scan_node_->mem_tracker());
     Status status = GetNextInternal(batch.get());
-    // Always add batch to the queue because it may contain data referenced by previously
-    // appended batches.
-    scan_node->AddMaterializedRowBatch(move(batch));
+    if (batch->num_rows() > 0) returned_rows = true;
+    // Always add batch to the queue if any rows were returned because it may contain
+    // data referenced by previously appended batches.
+    if (returned_rows) scan_node->AddMaterializedRowBatch(move(batch));
     RETURN_IF_ERROR(status);
   } while (!eos_ && !scan_node_->ReachedLimit());
   return Status::OK();

http://git-wip-us.apache.org/repos/asf/impala/blob/3c376950/be/src/exec/scan-node.h
----------------------------------------------------------------------
diff --git a/be/src/exec/scan-node.h b/be/src/exec/scan-node.h
index 6e137af..87d37c2 100644
--- a/be/src/exec/scan-node.h
+++ b/be/src/exec/scan-node.h
@@ -181,13 +181,15 @@ class ScanNode : public ExecNode {
   /// Aggregated scanner thread counters
   RuntimeProfile::ThreadCounters* scanner_thread_counters_;
 
-  /// The number of active scanner threads that are not blocked by IO.
+  /// The number of scanner threads currently running.
   RuntimeProfile::Counter active_scanner_thread_counter_;
 
   /// Average number of active scanner threads
   /// This should be created in Open and stopped when all the scanner threads are done.
   RuntimeProfile::Counter* average_scanner_thread_concurrency_;
 
+  /// Cumulative number of scanner threads created during the scan. Some may be created
+  /// and then destroyed, so this can exceed the peak number of threads.
   RuntimeProfile::Counter* num_scanner_threads_started_counter_;
 
   /// Expressions to evaluate the input rows for filtering against runtime filters.

http://git-wip-us.apache.org/repos/asf/impala/blob/3c376950/be/src/runtime/mem-pool.cc
----------------------------------------------------------------------
diff --git a/be/src/runtime/mem-pool.cc b/be/src/runtime/mem-pool.cc
index 3038185..8b6a4bf 100644
--- a/be/src/runtime/mem-pool.cc
+++ b/be/src/runtime/mem-pool.cc
@@ -184,11 +184,7 @@ void MemPool::AcquireData(MemPool* src, bool keep_current) {
   src->total_reserved_bytes_ -= total_transfered_bytes;
   total_reserved_bytes_ += total_transfered_bytes;
 
-  // Skip unnecessary atomic ops if the mem_trackers are the same.
-  if (src->mem_tracker_ != mem_tracker_) {
-    src->mem_tracker_->Release(total_transfered_bytes);
-    mem_tracker_->Consume(total_transfered_bytes);
-  }
+  src->mem_tracker_->TransferTo(mem_tracker_, total_transfered_bytes);
 
   // insert new chunks after current_chunk_idx_
   vector<ChunkInfo>::iterator insert_chunk = chunks_.begin() + current_chunk_idx_ + 1;
@@ -212,6 +208,11 @@ void MemPool::AcquireData(MemPool* src, bool keep_current) {
   DCHECK(CheckIntegrity(false));
 }
 
+void MemPool::SetMemTracker(MemTracker* new_tracker) {
+  mem_tracker_->TransferTo(new_tracker, total_reserved_bytes_);
+  mem_tracker_ = new_tracker;
+}
+
 string MemPool::DebugString() {
   stringstream out;
   char str[16];
@@ -244,8 +245,10 @@ bool MemPool::CheckIntegrity(bool check_current_chunk_empty) {
 
   // check that current_chunk_idx_ points to the last chunk with allocated data
   int64_t total_allocated = 0;
+  int64_t total_reserved = 0;
   for (int i = 0; i < chunks_.size(); ++i) {
     DCHECK_GT(chunks_[i].size, 0);
+    total_reserved += chunks_[i].size;
     if (i < current_chunk_idx_) {
       DCHECK_GT(chunks_[i].allocated_bytes, 0);
     } else if (i == current_chunk_idx_) {
@@ -257,5 +260,6 @@ bool MemPool::CheckIntegrity(bool check_current_chunk_empty) {
     total_allocated += chunks_[i].allocated_bytes;
   }
   DCHECK_EQ(total_allocated, total_allocated_bytes_);
+  DCHECK_EQ(total_reserved, total_reserved_bytes_);
   return true;
 }

http://git-wip-us.apache.org/repos/asf/impala/blob/3c376950/be/src/runtime/mem-pool.h
----------------------------------------------------------------------
diff --git a/be/src/runtime/mem-pool.h b/be/src/runtime/mem-pool.h
index b748f7f..2d12da8 100644
--- a/be/src/runtime/mem-pool.h
+++ b/be/src/runtime/mem-pool.h
@@ -155,6 +155,9 @@ class MemPool {
   /// All offsets handed out by calls to GetCurrentOffset() for 'src' become invalid.
   void AcquireData(MemPool* src, bool keep_current);
 
+  /// Change the MemTracker, updating consumption on the current and new tracker.
+  void SetMemTracker(MemTracker* new_tracker);
+
   std::string DebugString();
 
   int64_t total_allocated_bytes() const { return total_allocated_bytes_; }

http://git-wip-us.apache.org/repos/asf/impala/blob/3c376950/be/src/runtime/mem-tracker-test.cc
----------------------------------------------------------------------
diff --git a/be/src/runtime/mem-tracker-test.cc b/be/src/runtime/mem-tracker-test.cc
index faeb6a9..285f897 100644
--- a/be/src/runtime/mem-tracker-test.cc
+++ b/be/src/runtime/mem-tracker-test.cc
@@ -176,6 +176,67 @@ TEST(MemTestTest, TrackerHierarchy) {
   c2.Release(60);
 }
 
+// Test that we can transfer between MemTrackers without temporary double-counting
+// in ancestors
+TEST(MemTestTest, TransferTo) {
+  MemTracker root(100);
+  MemTracker parent(-1, "", &root);
+  MemTracker uncle(-1, "", &root);
+  MemTracker child1(-1, "", &parent);
+  MemTracker child2(-1, "", &parent);
+
+  child1.Consume(100);
+  // To self.
+  child1.TransferTo(&child1, 100);
+  EXPECT_EQ(child1.consumption(), 100);
+  EXPECT_EQ(child1.peak_consumption(), 100);
+  EXPECT_EQ(parent.consumption(), 100);
+  EXPECT_EQ(parent.peak_consumption(), 100);
+
+  // Child to parent.
+  child1.TransferTo(&parent, 100);
+  EXPECT_EQ(child1.consumption(), 0);
+  EXPECT_EQ(child1.peak_consumption(), 100);
+  EXPECT_EQ(parent.consumption(), 100);
+  EXPECT_EQ(parent.peak_consumption(), 100);
+
+  // Parent to child
+  parent.TransferTo(&child1, 100);
+  EXPECT_EQ(child1.consumption(), 100);
+  EXPECT_EQ(child1.peak_consumption(), 100);
+  EXPECT_EQ(parent.consumption(), 100);
+  EXPECT_EQ(parent.peak_consumption(), 100);
+
+  // Child to child.
+  child1.TransferTo(&child2, 50);
+  EXPECT_EQ(child1.consumption(), 50);
+  EXPECT_EQ(child2.consumption(), 50);
+  EXPECT_EQ(child2.peak_consumption(), 50);
+  EXPECT_EQ(parent.consumption(), 100);
+  EXPECT_EQ(parent.peak_consumption(), 100);
+
+  // Child to uncle.
+  child1.TransferTo(&uncle, 50);
+  EXPECT_EQ(child1.consumption(), 0);
+  EXPECT_EQ(uncle.consumption(), 50);
+  EXPECT_EQ(uncle.peak_consumption(), 50);
+  EXPECT_EQ(parent.consumption(), 50);
+  EXPECT_EQ(parent.peak_consumption(), 100);
+  EXPECT_EQ(root.consumption(), 100);
+  EXPECT_EQ(root.peak_consumption(), 100);
+
+  // Child to root
+  child2.TransferTo(&root, 50);
+  EXPECT_EQ(child2.consumption(), 0);
+  EXPECT_EQ(parent.consumption(), 0);
+  EXPECT_EQ(parent.peak_consumption(), 100);
+  EXPECT_EQ(root.consumption(), 100);
+  EXPECT_EQ(root.peak_consumption(), 100);
+
+  uncle.Release(50);
+  root.Release(50);
+}
+
 class GcFunctionHelper {
  public:
   static const int NUM_RELEASE_BYTES = 1;

http://git-wip-us.apache.org/repos/asf/impala/blob/3c376950/be/src/runtime/mem-tracker.cc
----------------------------------------------------------------------
diff --git a/be/src/runtime/mem-tracker.cc b/be/src/runtime/mem-tracker.cc
index 96c02a2..4762928 100644
--- a/be/src/runtime/mem-tracker.cc
+++ b/be/src/runtime/mem-tracker.cc
@@ -223,6 +223,25 @@ void MemTracker::RegisterMetrics(MetricGroup* metrics, const string& prefix) {
   limit_metric_ = metrics->AddGauge(Substitute("$0.limit", prefix), limit_);
 }
 
+void MemTracker::TransferTo(MemTracker* dst, int64_t bytes) {
+  DCHECK_EQ(all_trackers_.back(), dst->all_trackers_.back())
+      << "Must have same root";
+  // Find the common ancestor and update trackers between 'this'/'dst' and
+  // the common ancestor. This logic handles all cases, including the
+  // two trackers being the same or being ancestors of each other because
+  // 'all_trackers_' includes the current tracker.
+  int ancestor_idx = all_trackers_.size() - 1;
+  int dst_ancestor_idx = dst->all_trackers_.size() - 1;
+  while (ancestor_idx > 0 && dst_ancestor_idx > 0
+      && all_trackers_[ancestor_idx - 1] == dst->all_trackers_[dst_ancestor_idx - 1]) {
+    --ancestor_idx;
+    --dst_ancestor_idx;
+  }
+  MemTracker* common_ancestor = all_trackers_[ancestor_idx];
+  ReleaseLocal(bytes, common_ancestor);
+  dst->ConsumeLocal(bytes, common_ancestor);
+}
+
 // Calling this on the query tracker results in output like:
 //
 //  Query(4a4c81fedaed337d:4acadfda00000000) Limit=10.00 GB Total=508.28 MB Peak=508.45 MB

http://git-wip-us.apache.org/repos/asf/impala/blob/3c376950/be/src/runtime/mem-tracker.h
----------------------------------------------------------------------
diff --git a/be/src/runtime/mem-tracker.h b/be/src/runtime/mem-tracker.h
index 8d28f8f..9c41693 100644
--- a/be/src/runtime/mem-tracker.h
+++ b/be/src/runtime/mem-tracker.h
@@ -234,6 +234,12 @@ class MemTracker {
     }
   }
 
+  /// Transfer 'bytes' of consumption from this tracker to 'dst', updating
+  /// all ancestors up to the first shared ancestor. Must not be used if
+  /// 'dst' has a limit, or an ancestor with a limit, that is not a common
+  /// ancestor with the tracker, because this does not check memory limits.
+  void TransferTo(MemTracker* dst, int64_t bytes);
+
   /// Returns true if a valid limit of this tracker or one of its ancestors is
   /// exceeded.
   bool AnyLimitExceeded() {

http://git-wip-us.apache.org/repos/asf/impala/blob/3c376950/be/src/runtime/row-batch.cc
----------------------------------------------------------------------
diff --git a/be/src/runtime/row-batch.cc b/be/src/runtime/row-batch.cc
index 3581bd9..ca2a9f3 100644
--- a/be/src/runtime/row-batch.cc
+++ b/be/src/runtime/row-batch.cc
@@ -455,6 +455,12 @@ void RowBatch::TransferResourceOwnership(RowBatch* dest) {
   Reset();
 }
 
+void RowBatch::SetMemTracker(MemTracker* new_tracker) {
+  tuple_data_pool_.SetMemTracker(new_tracker);
+  mem_tracker_->TransferTo(new_tracker, tuple_ptrs_size_);
+  mem_tracker_ = new_tracker;
+}
+
 int64_t RowBatch::GetDeserializedSize(const TRowBatch& batch) {
   return batch.uncompressed_size + batch.tuple_offsets.size() * sizeof(Tuple*);
 }

http://git-wip-us.apache.org/repos/asf/impala/blob/3c376950/be/src/runtime/row-batch.h
----------------------------------------------------------------------
diff --git a/be/src/runtime/row-batch.h b/be/src/runtime/row-batch.h
index d8cbe7c..08df8a8 100644
--- a/be/src/runtime/row-batch.h
+++ b/be/src/runtime/row-batch.h
@@ -315,6 +315,9 @@ class RowBatch {
   /// pool and buffers.
   void TransferResourceOwnership(RowBatch* dest);
 
+  /// Update accounting so that attached memory is accounted against 'new_tracker'.
+  void SetMemTracker(MemTracker* new_tracker);
+
   void CopyRow(TupleRow* src, TupleRow* dest) {
     memcpy(dest, src, num_tuples_per_row_ * sizeof(Tuple*));
   }