You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by ta...@apache.org on 2018/12/07 15:45:55 UTC

[2/2] impala git commit: IMPALA-1048: show sinks in exec summary

IMPALA-1048: show sinks in exec summary

The exec summary now includes the total time taken and memory
consumed by the data sink at the root of each fragment. Previously
the exec summary could hide where time and memory went while
executing a query.

The high-level changes are:
* Generalising logic in the exec summary and runtime profile to
  handle data sinks, not just plan nodes, including adding richer
  metadata to runtime profile nodes.
* Threading through metadata about the data sinks, like names and
  estimates, so that it can appear in the exec summary.

The major potential downside is that the new timings reported for
data stream sender can overlap with the receiver's time and
potentially cause confusion.

[localhost:21000] default> select count(distinct l_comment) from tpch_parquet.lineitem; summary;
Query: select count(distinct l_comment) from tpch_parquet.lineitem
Query submitted at: 2018-11-20 16:47:03 (Coordinator: http://tarmstrong-box:25000)
Query progress can be monitored at: http://tarmstrong-box:25000/query_plan?query_id=f5464383a3bb6878:54b5252b00000000
+---------------------------+
| count(distinct l_comment) |
+---------------------------+
| 4580667                   |
+---------------------------+
Fetched 1 row(s) in 4.53s
+---------------------+--------+----------+----------+-------+------------+-----------+---------------+-----------------------+
| Operator            | #Hosts | Avg Time | Max Time | #Rows | Est. #Rows | Peak Mem  | Est. Peak Mem | Detail                |
+---------------------+--------+----------+----------+-------+------------+-----------+---------------+-----------------------+
| F02:ROOT            | 1      | 50.56us  | 50.56us  |       |            | 0 B       | 0 B           |                       |
| 06:AGGREGATE        | 1      | 275.89us | 275.89us | 1     | 1          | 16.00 KB  | 10.00 MB      | FINALIZE              |
| 05:EXCHANGE         | 1      | 49.08us  | 49.08us  | 3     | 1          | 32.00 KB  | 16.00 KB      | UNPARTITIONED         |
| F01:EXCHANGE SENDER | 3      | 100.06us | 113.49us |       |            | 16.00 KB  | 0 B           |                       |
| 02:AGGREGATE        | 3      | 19.32ms  | 19.57ms  | 3     | 1          | 16.00 KB  | 10.00 MB      |                       |
| 04:AGGREGATE        | 3      | 1.29s    | 1.43s    | 4.58M | 4.65M      | 98.02 MB  | 62.63 MB      |                       |
| 03:EXCHANGE         | 3      | 241.64ms | 246.54ms | 5.01M | 4.65M      | 9.05 MB   | 10.12 MB      | HASH(l_comment)       |
| F00:EXCHANGE SENDER | 3      | 2.43s    | 2.58s    |       |            | 337.53 KB | 0 B           |                       |
| 01:AGGREGATE        | 3      | 1.26s    | 1.46s    | 5.01M | 4.65M      | 97.20 MB  | 121.17 MB     | STREAMING             |
| 00:SCAN HDFS        | 3      | 39.87ms  | 41.36ms  | 6.00M | 6.00M      | 27.87 MB  | 80.00 MB      | tpch_parquet.lineitem |
+---------------------+--------+----------+----------+-------+------------+-----------+---------------+-----------------------+

Testing:
Added a basic observability test.

Change-Id: I3fdf7bacae8ff597b255da65af453e174ba53544
Reviewed-on: http://gerrit.cloudera.org:8080/11967
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>


Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/d8792c21
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/d8792c21
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/d8792c21

Branch: refs/heads/master
Commit: d8792c21c5a385fc80e3cfe34293abc560c09b37
Parents: a18809b
Author: Tim Armstrong <ta...@cloudera.com>
Authored: Mon Nov 19 20:45:14 2018 -0800
Committer: Impala Public Jenkins <im...@cloudera.com>
Committed: Fri Dec 7 07:33:01 2018 +0000

----------------------------------------------------------------------
 be/src/exec/data-sink.cc                        | 30 +++++++----
 be/src/exec/data-sink.h                         |  6 ++-
 be/src/exec/exec-node.cc                        | 10 ++--
 be/src/exec/exec-node.h                         |  3 --
 be/src/exec/hbase-table-sink.cc                 |  6 +--
 be/src/exec/hbase-table-sink.h                  |  4 +-
 be/src/exec/hdfs-table-sink.cc                  |  6 +--
 be/src/exec/hdfs-table-sink.h                   |  4 +-
 be/src/exec/kudu-table-sink.cc                  |  6 +--
 be/src/exec/kudu-table-sink.h                   |  4 +-
 be/src/exec/nested-loop-join-builder.cc         |  2 +-
 be/src/exec/partitioned-hash-join-builder.cc    |  2 +-
 be/src/exec/plan-root-sink.cc                   | 15 ++++--
 be/src/exec/plan-root-sink.h                    |  2 +-
 be/src/runtime/coordinator-backend-state.cc     | 48 ++++++++---------
 be/src/runtime/coordinator.cc                   | 32 +++++++++--
 be/src/runtime/coordinator.h                    |  3 ++
 be/src/runtime/data-stream-test.cc              |  2 +-
 be/src/runtime/krpc-data-stream-sender.cc       |  9 ++--
 be/src/runtime/krpc-data-stream-sender.h        |  2 +-
 be/src/util/runtime-profile.cc                  | 32 +++++++----
 be/src/util/runtime-profile.h                   | 18 ++++---
 be/src/util/summary-util.cc                     | 13 +++--
 common/thrift/DataSinks.thrift                  |  8 +++
 common/thrift/ExecStats.thrift                  |  9 ++--
 common/thrift/RuntimeProfile.thrift             | 18 ++++++-
 common/thrift/Types.thrift                      |  1 +
 .../org/apache/impala/planner/DataSink.java     | 30 ++++++++++-
 .../apache/impala/planner/DataStreamSink.java   | 16 ++++--
 .../apache/impala/planner/HBaseTableSink.java   | 17 ++++--
 .../apache/impala/planner/HdfsTableSink.java    | 16 ++++--
 .../apache/impala/planner/JoinBuildSink.java    | 16 ++++--
 .../apache/impala/planner/KuduTableSink.java    | 16 ++++--
 .../org/apache/impala/planner/PlanRootSink.java | 13 ++++-
 shell/impala_client.py                          |  5 +-
 tests/beeswax/impala_beeswax.py                 |  5 +-
 tests/query_test/test_observability.py          | 57 +++++++++++++++++---
 37 files changed, 349 insertions(+), 137 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/impala/blob/d8792c21/be/src/exec/data-sink.cc
----------------------------------------------------------------------
diff --git a/be/src/exec/data-sink.cc b/be/src/exec/data-sink.cc
index 4071f43..45a5b2e 100644
--- a/be/src/exec/data-sink.cc
+++ b/be/src/exec/data-sink.cc
@@ -48,9 +48,15 @@ namespace impala {
 // Empty string
 const char* const DataSink::ROOT_PARTITION_KEY = "";
 
-DataSink::DataSink(const RowDescriptor* row_desc, const string& name, RuntimeState* state)
+DataSink::DataSink(TDataSinkId sink_id, const RowDescriptor* row_desc, const string& name,
+    RuntimeState* state)
   : closed_(false), row_desc_(row_desc), name_(name) {
   profile_ = RuntimeProfile::Create(state->obj_pool(), name);
+  if (sink_id != -1) {
+    // There is one sink per fragment so we can use the fragment index as a unique
+    // identifier.
+    profile_->SetDataSinkId(sink_id);
+  }
 }
 
 DataSink::~DataSink() {
@@ -63,27 +69,31 @@ Status DataSink::Create(const TPlanFragmentCtx& fragment_ctx,
   const TDataSink& thrift_sink = fragment_ctx.fragment.output_sink;
   const vector<TExpr>& thrift_output_exprs = fragment_ctx.fragment.output_exprs;
   ObjectPool* pool = state->obj_pool();
+  // We have one fragment per sink, so we can use the fragment index as the sink ID.
+  TDataSinkId sink_id = fragment_ctx.fragment.idx;
   switch (thrift_sink.type) {
     case TDataSinkType::DATA_STREAM_SINK:
       if (!thrift_sink.__isset.stream_sink) return Status("Missing data stream sink.");
-
       // TODO: figure out good buffer size based on size of output row
-      *sink = pool->Add(new KrpcDataStreamSender(fragment_instance_ctx.sender_id,
-          row_desc, thrift_sink.stream_sink, fragment_ctx.destinations,
-          FLAGS_data_stream_sender_buffer_size, state));
+      *sink = pool->Add(new KrpcDataStreamSender(sink_id,
+          fragment_instance_ctx.sender_id, row_desc, thrift_sink.stream_sink,
+          fragment_ctx.destinations, FLAGS_data_stream_sender_buffer_size, state));
       break;
     case TDataSinkType::TABLE_SINK:
       if (!thrift_sink.__isset.table_sink) return Status("Missing table sink.");
       switch (thrift_sink.table_sink.type) {
         case TTableSinkType::HDFS:
-          *sink = pool->Add(new HdfsTableSink(row_desc, thrift_sink, state));
+          *sink =
+              pool->Add(new HdfsTableSink(sink_id, row_desc, thrift_sink, state));
           break;
         case TTableSinkType::HBASE:
-          *sink = pool->Add(new HBaseTableSink(row_desc, thrift_sink, state));
+          *sink =
+              pool->Add(new HBaseTableSink(sink_id, row_desc, thrift_sink, state));
           break;
         case TTableSinkType::KUDU:
           RETURN_IF_ERROR(CheckKuduAvailability());
-          *sink = pool->Add(new KuduTableSink(row_desc, thrift_sink, state));
+          *sink =
+              pool->Add(new KuduTableSink(sink_id, row_desc, thrift_sink, state));
           break;
         default:
           stringstream error_msg;
@@ -96,8 +106,10 @@ Status DataSink::Create(const TPlanFragmentCtx& fragment_ctx,
       }
       break;
     case TDataSinkType::PLAN_ROOT_SINK:
-      *sink = pool->Add(new PlanRootSink(row_desc, state));
+      *sink = pool->Add(new PlanRootSink(sink_id, row_desc, state));
       break;
+    case TDataSinkType::JOIN_BUILD_SINK:
+      // IMPALA-4224 - join build sink not supported in backend execution.
     default:
       stringstream error_msg;
       map<int, const char*>::const_iterator i =

http://git-wip-us.apache.org/repos/asf/impala/blob/d8792c21/be/src/exec/data-sink.h
----------------------------------------------------------------------
diff --git a/be/src/exec/data-sink.h b/be/src/exec/data-sink.h
index cc5e6ba..60968c4 100644
--- a/be/src/exec/data-sink.h
+++ b/be/src/exec/data-sink.h
@@ -54,7 +54,11 @@ class TInsertStats;
 /// Close() is called to release any resources before destroying the sink.
 class DataSink {
  public:
-  DataSink(const RowDescriptor* row_desc, const string& name, RuntimeState* state);
+  /// If this is the sink at the root of a fragment, 'sink_id' must be a unique ID for
+  /// the sink for use in runtime profiles and other purposes. Otherwise this is a join
+  /// build sink owned by an ExecNode and 'sink_id' must be -1.
+  DataSink(TDataSinkId sink_id, const RowDescriptor* row_desc, const string& name,
+      RuntimeState* state);
   virtual ~DataSink();
 
   /// Setup. Call before Send(), Open(), or Close() during the prepare phase of the query

http://git-wip-us.apache.org/repos/asf/impala/blob/d8792c21/be/src/exec/exec-node.cc
----------------------------------------------------------------------
diff --git a/be/src/exec/exec-node.cc b/be/src/exec/exec-node.cc
index f9ed145..16f074a 100644
--- a/be/src/exec/exec-node.cc
+++ b/be/src/exec/exec-node.cc
@@ -74,10 +74,6 @@ namespace impala {
 
 const string ExecNode::ROW_THROUGHPUT_COUNTER = "RowsReturnedRate";
 
-int ExecNode::GetNodeIdFromProfile(RuntimeProfile* p) {
-  return p->metadata();
-}
-
 ExecNode::ExecNode(ObjectPool* pool, const TPlanNode& tnode, const DescriptorTbl& descs)
   : id_(tnode.node_id),
     type_(tnode.node_type),
@@ -86,14 +82,14 @@ ExecNode::ExecNode(ObjectPool* pool, const TPlanNode& tnode, const DescriptorTbl
     resource_profile_(tnode.resource_profile),
     limit_(tnode.limit),
     num_rows_returned_(0),
-    runtime_profile_(RuntimeProfile::Create(pool_,
-        Substitute("$0 (id=$1)", PrintThriftEnum(tnode.node_type), id_))),
+    runtime_profile_(RuntimeProfile::Create(
+        pool_, Substitute("$0 (id=$1)", PrintThriftEnum(tnode.node_type), id_))),
     rows_returned_counter_(NULL),
     rows_returned_rate_(NULL),
     containing_subplan_(NULL),
     disable_codegen_(tnode.disable_codegen),
     is_closed_(false) {
-  runtime_profile_->set_metadata(id_);
+  runtime_profile_->SetPlanNodeId(id_);
   debug_options_.phase = TExecNodePhase::INVALID;
 }
 

http://git-wip-us.apache.org/repos/asf/impala/blob/d8792c21/be/src/exec/exec-node.h
----------------------------------------------------------------------
diff --git a/be/src/exec/exec-node.h b/be/src/exec/exec-node.h
index 89b9ca1..28a0baa 100644
--- a/be/src/exec/exec-node.h
+++ b/be/src/exec/exec-node.h
@@ -219,9 +219,6 @@ class ExecNode {
   /// check to see if codegen was enabled for the enclosing fragment.
   bool IsNodeCodegenDisabled() const;
 
-  /// Extract node id from p->name().
-  static int GetNodeIdFromProfile(RuntimeProfile* p);
-
   /// Returns true if this node is inside the right-hand side plan tree of a SubplanNode.
   /// Valid to call in or after Prepare().
   bool IsInSubplan() const { return containing_subplan_ != NULL; }

http://git-wip-us.apache.org/repos/asf/impala/blob/d8792c21/be/src/exec/hbase-table-sink.cc
----------------------------------------------------------------------
diff --git a/be/src/exec/hbase-table-sink.cc b/be/src/exec/hbase-table-sink.cc
index 4f54d6e..c907bcc 100644
--- a/be/src/exec/hbase-table-sink.cc
+++ b/be/src/exec/hbase-table-sink.cc
@@ -31,9 +31,9 @@
 
 namespace impala {
 
-HBaseTableSink::HBaseTableSink(const RowDescriptor* row_desc, const TDataSink& tsink,
-    RuntimeState* state)
-  : DataSink(row_desc, "HBaseTableSink", state),
+HBaseTableSink::HBaseTableSink(TDataSinkId sink_id, const RowDescriptor* row_desc,
+    const TDataSink& tsink, RuntimeState* state)
+  : DataSink(sink_id, row_desc, "HBaseTableSink", state),
     table_id_(tsink.table_sink.target_table_id),
     table_desc_(NULL),
     hbase_table_writer_(NULL) {

http://git-wip-us.apache.org/repos/asf/impala/blob/d8792c21/be/src/exec/hbase-table-sink.h
----------------------------------------------------------------------
diff --git a/be/src/exec/hbase-table-sink.h b/be/src/exec/hbase-table-sink.h
index f8075ae..d973ad4 100644
--- a/be/src/exec/hbase-table-sink.h
+++ b/be/src/exec/hbase-table-sink.h
@@ -35,8 +35,8 @@ namespace impala {
 /// eventually be written into an HBase table.
 class HBaseTableSink : public DataSink {
  public:
-  HBaseTableSink(const RowDescriptor* row_desc, const TDataSink& tsink,
-      RuntimeState* state);
+  HBaseTableSink(TDataSinkId sink_id, const RowDescriptor* row_desc,
+      const TDataSink& tsink, RuntimeState* state);
   virtual Status Prepare(RuntimeState* state, MemTracker* parent_mem_tracker);
   virtual Status Send(RuntimeState* state, RowBatch* batch);
   virtual Status FlushFinal(RuntimeState* state);

http://git-wip-us.apache.org/repos/asf/impala/blob/d8792c21/be/src/exec/hdfs-table-sink.cc
----------------------------------------------------------------------
diff --git a/be/src/exec/hdfs-table-sink.cc b/be/src/exec/hdfs-table-sink.cc
index 6d2bf00..23993e8 100644
--- a/be/src/exec/hdfs-table-sink.cc
+++ b/be/src/exec/hdfs-table-sink.cc
@@ -52,9 +52,9 @@ using namespace strings;
 
 namespace impala {
 
-HdfsTableSink::HdfsTableSink(const RowDescriptor* row_desc, const TDataSink& tsink,
-    RuntimeState* state)
-  : DataSink(row_desc, "HdfsTableSink", state),
+HdfsTableSink::HdfsTableSink(TDataSinkId sink_id, const RowDescriptor* row_desc,
+    const TDataSink& tsink, RuntimeState* state)
+  : DataSink(sink_id, row_desc, "HdfsTableSink", state),
     table_desc_(nullptr),
     prototype_partition_(nullptr),
     table_id_(tsink.table_sink.target_table_id),

http://git-wip-us.apache.org/repos/asf/impala/blob/d8792c21/be/src/exec/hdfs-table-sink.h
----------------------------------------------------------------------
diff --git a/be/src/exec/hdfs-table-sink.h b/be/src/exec/hdfs-table-sink.h
index 002e866..277a212 100644
--- a/be/src/exec/hdfs-table-sink.h
+++ b/be/src/exec/hdfs-table-sink.h
@@ -127,8 +127,8 @@ struct OutputPartition {
 /// This is consistent with Hive's behavior.
 class HdfsTableSink : public DataSink {
  public:
-  HdfsTableSink(const RowDescriptor* row_desc, const TDataSink& tsink,
-      RuntimeState* state);
+  HdfsTableSink(TDataSinkId sink_id, const RowDescriptor* row_desc,
+      const TDataSink& tsink, RuntimeState* state);
 
   /// Prepares output_exprs and partition_key_exprs, and connects to HDFS.
   virtual Status Prepare(RuntimeState* state, MemTracker* parent_mem_tracker);

http://git-wip-us.apache.org/repos/asf/impala/blob/d8792c21/be/src/exec/kudu-table-sink.cc
----------------------------------------------------------------------
diff --git a/be/src/exec/kudu-table-sink.cc b/be/src/exec/kudu-table-sink.cc
index b50394d..954c65b 100644
--- a/be/src/exec/kudu-table-sink.cc
+++ b/be/src/exec/kudu-table-sink.cc
@@ -66,9 +66,9 @@ namespace impala {
 // Send 7MB buffers to Kudu, matching a hard-coded size in Kudu (KUDU-1693).
 const static int INDIVIDUAL_BUFFER_SIZE = 7 * 1024 * 1024;
 
-KuduTableSink::KuduTableSink(const RowDescriptor* row_desc, const TDataSink& tsink,
-    RuntimeState* state)
-  : DataSink(row_desc, "KuduTableSink", state),
+KuduTableSink::KuduTableSink(TDataSinkId sink_id, const RowDescriptor* row_desc,
+    const TDataSink& tsink, RuntimeState* state)
+  : DataSink(sink_id, row_desc, "KuduTableSink", state),
     table_id_(tsink.table_sink.target_table_id),
     sink_action_(tsink.table_sink.action),
     kudu_table_sink_(tsink.table_sink.kudu_table_sink),

http://git-wip-us.apache.org/repos/asf/impala/blob/d8792c21/be/src/exec/kudu-table-sink.h
----------------------------------------------------------------------
diff --git a/be/src/exec/kudu-table-sink.h b/be/src/exec/kudu-table-sink.h
index 3915d0b..ad6c560 100644
--- a/be/src/exec/kudu-table-sink.h
+++ b/be/src/exec/kudu-table-sink.h
@@ -53,8 +53,8 @@ namespace impala {
 /// status. All reported errors (ignored or not) will be logged via the RuntimeState.
 class KuduTableSink : public DataSink {
  public:
-  KuduTableSink(const RowDescriptor* row_desc, const TDataSink& tsink,
-      RuntimeState* state);
+  KuduTableSink(TDataSinkId sink_id, const RowDescriptor* row_desc,
+      const TDataSink& tsink, RuntimeState* state);
 
   /// Prepares the expressions to be applied and creates a KuduSchema based on the
   /// expressions and KuduTableDescriptor.

http://git-wip-us.apache.org/repos/asf/impala/blob/d8792c21/be/src/exec/nested-loop-join-builder.cc
----------------------------------------------------------------------
diff --git a/be/src/exec/nested-loop-join-builder.cc b/be/src/exec/nested-loop-join-builder.cc
index 31e7a8d..a521fa9 100644
--- a/be/src/exec/nested-loop-join-builder.cc
+++ b/be/src/exec/nested-loop-join-builder.cc
@@ -28,7 +28,7 @@
 using namespace impala;
 
 NljBuilder::NljBuilder(const RowDescriptor* row_desc, RuntimeState* state)
-  : DataSink(row_desc, "Nested Loop Join Builder", state),
+  : DataSink(-1, row_desc, "Nested Loop Join Builder", state),
     build_batch_cache_(row_desc, state->batch_size()) {}
 
 Status NljBuilder::Prepare(RuntimeState* state, MemTracker* parent_mem_tracker) {

http://git-wip-us.apache.org/repos/asf/impala/blob/d8792c21/be/src/exec/partitioned-hash-join-builder.cc
----------------------------------------------------------------------
diff --git a/be/src/exec/partitioned-hash-join-builder.cc b/be/src/exec/partitioned-hash-join-builder.cc
index 2232108..b21a033 100644
--- a/be/src/exec/partitioned-hash-join-builder.cc
+++ b/be/src/exec/partitioned-hash-join-builder.cc
@@ -55,7 +55,7 @@ PhjBuilder::PhjBuilder(int join_node_id, TJoinOp::type join_op,
     const RowDescriptor* probe_row_desc, const RowDescriptor* build_row_desc,
     RuntimeState* state, BufferPool::ClientHandle* buffer_pool_client,
     int64_t spillable_buffer_size, int64_t max_row_buffer_size)
-  : DataSink(build_row_desc,
+  : DataSink(-1, build_row_desc,
         Substitute("Hash Join Builder (join_node_id=$0)", join_node_id), state),
     runtime_state_(state),
     join_node_id_(join_node_id),

http://git-wip-us.apache.org/repos/asf/impala/blob/d8792c21/be/src/exec/plan-root-sink.cc
----------------------------------------------------------------------
diff --git a/be/src/exec/plan-root-sink.cc b/be/src/exec/plan-root-sink.cc
index 1f5b2e5..4bb9693 100644
--- a/be/src/exec/plan-root-sink.cc
+++ b/be/src/exec/plan-root-sink.cc
@@ -32,8 +32,9 @@ using boost::mutex;
 
 namespace impala {
 
-PlanRootSink::PlanRootSink(const RowDescriptor* row_desc, RuntimeState* state)
-  : DataSink(row_desc, "PLAN_ROOT_SINK", state) {}
+PlanRootSink::PlanRootSink(
+    TDataSinkId sink_id, const RowDescriptor* row_desc, RuntimeState* state)
+  : DataSink(sink_id, row_desc, "PLAN_ROOT_SINK", state) {}
 
 namespace {
 
@@ -61,9 +62,10 @@ void ValidateCollectionSlots(const RowDescriptor& row_desc, RowBatch* batch) {
   }
 #endif
 }
-}
+} // namespace
 
 Status PlanRootSink::Send(RuntimeState* state, RowBatch* batch) {
+  SCOPED_TIMER(profile()->total_time_counter());
   ValidateCollectionSlots(*row_desc_, batch);
   int current_batch_row = 0;
 
@@ -74,7 +76,10 @@ Status PlanRootSink::Send(RuntimeState* state, RowBatch* batch) {
     unique_lock<mutex> l(lock_);
     // Wait until the consumer gives us a result set to fill in, or the fragment
     // instance has been cancelled.
-    while (results_ == nullptr && !state->is_cancelled()) sender_cv_.Wait(l);
+    while (results_ == nullptr && !state->is_cancelled()) {
+      SCOPED_TIMER(profile_->inactive_timer());
+      sender_cv_.Wait(l);
+    }
     RETURN_IF_CANCELLED(state);
 
     // Otherwise the consumer is ready. Fill out the rows.
@@ -94,6 +99,7 @@ Status PlanRootSink::Send(RuntimeState* state, RowBatch* batch) {
 }
 
 Status PlanRootSink::FlushFinal(RuntimeState* state) {
+  SCOPED_TIMER(profile()->total_time_counter());
   unique_lock<mutex> l(lock_);
   sender_state_ = SenderState::EOS;
   consumer_cv_.NotifyAll();
@@ -101,6 +107,7 @@ Status PlanRootSink::FlushFinal(RuntimeState* state) {
 }
 
 void PlanRootSink::Close(RuntimeState* state) {
+  SCOPED_TIMER(profile()->total_time_counter());
   unique_lock<mutex> l(lock_);
   // FlushFinal() won't have been called when the fragment instance encounters an error
   // before sending all rows.

http://git-wip-us.apache.org/repos/asf/impala/blob/d8792c21/be/src/exec/plan-root-sink.h
----------------------------------------------------------------------
diff --git a/be/src/exec/plan-root-sink.h b/be/src/exec/plan-root-sink.h
index 300c993..b68fff7 100644
--- a/be/src/exec/plan-root-sink.h
+++ b/be/src/exec/plan-root-sink.h
@@ -62,7 +62,7 @@ class ScalarExprEvaluator;
 /// and consumer. See IMPALA-4268.
 class PlanRootSink : public DataSink {
  public:
-  PlanRootSink(const RowDescriptor* row_desc, RuntimeState* state);
+  PlanRootSink(TDataSinkId sink_id, const RowDescriptor* row_desc, RuntimeState* state);
 
   /// Sends a new batch. Ownership of 'batch' remains with the sender. Blocks until the
   /// consumer has consumed 'batch' by calling GetNext().

http://git-wip-us.apache.org/repos/asf/impala/blob/d8792c21/be/src/runtime/coordinator-backend-state.cc
----------------------------------------------------------------------
diff --git a/be/src/runtime/coordinator-backend-state.cc b/be/src/runtime/coordinator-backend-state.cc
index 252eb0c..6a09b7b 100644
--- a/be/src/runtime/coordinator-backend-state.cc
+++ b/be/src/runtime/coordinator-backend-state.cc
@@ -475,17 +475,13 @@ Coordinator::BackendState::InstanceStats::InstanceStats(
 void Coordinator::BackendState::InstanceStats::InitCounters() {
   vector<RuntimeProfile*> children;
   profile_->GetAllChildren(&children);
-  for (RuntimeProfile* p: children) {
-    PlanNodeId id = ExecNode::GetNodeIdFromProfile(p);
+  for (RuntimeProfile* p : children) {
     // This profile is not for an exec node.
-    if (id == g_ImpalaInternalService_constants.INVALID_PLAN_NODE_ID) continue;
-
-    RuntimeProfile::Counter* c =
-        p->GetCounter(ScanNode::SCAN_RANGES_COMPLETE_COUNTER);
+    if (!p->metadata().__isset.plan_node_id) continue;
+    RuntimeProfile::Counter* c = p->GetCounter(ScanNode::SCAN_RANGES_COMPLETE_COUNTER);
     if (c != nullptr) scan_ranges_complete_counters_.push_back(c);
 
-    RuntimeProfile::Counter* bytes_read =
-        p->GetCounter(ScanNode::BYTES_READ_COUNTER);
+    RuntimeProfile::Counter* bytes_read = p->GetCounter(ScanNode::BYTES_READ_COUNTER);
     if (bytes_read != nullptr) bytes_read_counters_.push_back(bytes_read);
   }
 }
@@ -512,30 +508,34 @@ void Coordinator::BackendState::InstanceStats::Update(
   vector<RuntimeProfile*> children;
   profile_->GetAllChildren(&children);
   TExecSummary& thrift_exec_summary = exec_summary->thrift_exec_summary;
-  for (RuntimeProfile* child: children) {
-    int node_id = ExecNode::GetNodeIdFromProfile(child);
-    if (node_id == -1) continue;
-
-    // TODO: create plan_node_id_to_summary_map_
-    TPlanNodeExecSummary& node_exec_summary =
-        thrift_exec_summary.nodes[exec_summary->node_id_to_idx_map[node_id]];
+  for (RuntimeProfile* child : children) {
+    bool is_plan_node = child->metadata().__isset.plan_node_id;
+    bool is_data_sink = child->metadata().__isset.data_sink_id;
+    // Plan Nodes and data sinks get an entry in the summary.
+    if (!is_plan_node && !is_data_sink) continue;
+
+    int exec_summary_idx;
+    if (is_plan_node) {
+      exec_summary_idx = exec_summary->node_id_to_idx_map[child->metadata().plan_node_id];
+    } else {
+      exec_summary_idx =
+          exec_summary->data_sink_id_to_idx_map[child->metadata().data_sink_id];
+    }
+    TPlanNodeExecSummary& exec_summary = thrift_exec_summary.nodes[exec_summary_idx];
     int per_fragment_instance_idx = exec_params_.per_fragment_instance_idx;
-    DCHECK_LT(per_fragment_instance_idx, node_exec_summary.exec_stats.size())
-        << " node_id=" << node_id << " instance_id="
-        << PrintId(exec_params_.instance_id)
+    DCHECK_LT(per_fragment_instance_idx, exec_summary.exec_stats.size())
+        << " name=" << child->name()
+        << " instance_id=" << PrintId(exec_params_.instance_id)
         << " fragment_idx=" << exec_params_.fragment().idx;
-    TExecStats& instance_stats =
-        node_exec_summary.exec_stats[per_fragment_instance_idx];
+    TExecStats& instance_stats = exec_summary.exec_stats[per_fragment_instance_idx];
 
     RuntimeProfile::Counter* rows_counter = child->GetCounter("RowsReturned");
     RuntimeProfile::Counter* mem_counter = child->GetCounter("PeakMemoryUsage");
-    if (rows_counter != nullptr) {
-      instance_stats.__set_cardinality(rows_counter->value());
-    }
+    if (rows_counter != nullptr) instance_stats.__set_cardinality(rows_counter->value());
     if (mem_counter != nullptr) instance_stats.__set_memory_used(mem_counter->value());
     instance_stats.__set_latency_ns(child->local_time());
     // TODO: track interesting per-node metrics
-    node_exec_summary.__isset.exec_stats = true;
+    exec_summary.__isset.exec_stats = true;
   }
 
   // determine newly-completed scan ranges and update scan_range_progress

http://git-wip-us.apache.org/repos/asf/impala/blob/d8792c21/be/src/runtime/coordinator.cc
----------------------------------------------------------------------
diff --git a/be/src/runtime/coordinator.cc b/be/src/runtime/coordinator.cc
index 9dfb615..7d0c0e0 100644
--- a/be/src/runtime/coordinator.cc
+++ b/be/src/runtime/coordinator.cc
@@ -223,9 +223,34 @@ void Coordinator::ExecSummary::Init(const QuerySchedule& schedule) {
       int root_node_idx = thrift_exec_summary.nodes.size();
 
       const TPlan& plan = fragment.plan;
+      const TDataSink& output_sink = fragment.output_sink;
       int num_instances =
           schedule.GetFragmentExecParams(fragment.idx).instance_exec_params.size();
-      for (const TPlanNode& node: plan.nodes) {
+
+      // Add the data sink at the root of the fragment.
+      data_sink_id_to_idx_map[fragment.idx] = thrift_exec_summary.nodes.size();
+      thrift_exec_summary.nodes.emplace_back();
+      // Note that some clients like impala-shell depend on many of these fields being
+      // set, even if they are optional in the thrift.
+      TPlanNodeExecSummary& node_summary = thrift_exec_summary.nodes.back();
+      node_summary.__set_node_id(-1);
+      node_summary.__set_fragment_idx(fragment.idx);
+      node_summary.__set_label(output_sink.label);
+      node_summary.__set_label_detail("");
+      node_summary.__set_num_children(1);
+      DCHECK(output_sink.__isset.estimated_stats);
+      node_summary.__set_estimated_stats(output_sink.estimated_stats);
+      node_summary.exec_stats.resize(num_instances);
+
+      // We don't track rows returned from sinks, but some clients like impala-shell
+      // expect it to be set in the thrift struct. Set it to -1 for compatibility
+      // with those tools.
+      node_summary.estimated_stats.__set_cardinality(-1);
+      for (TExecStats& instance_stats : node_summary.exec_stats) {
+        instance_stats.__set_cardinality(-1);
+      }
+
+      for (const TPlanNode& node : plan.nodes) {
         node_id_to_idx_map[node.node_id] = thrift_exec_summary.nodes.size();
         thrift_exec_summary.nodes.emplace_back();
         TPlanNodeExecSummary& node_summary = thrift_exec_summary.nodes.back();
@@ -234,9 +259,8 @@ void Coordinator::ExecSummary::Init(const QuerySchedule& schedule) {
         node_summary.__set_label(node.label);
         node_summary.__set_label_detail(node.label_detail);
         node_summary.__set_num_children(node.num_children);
-        if (node.__isset.estimated_stats) {
-          node_summary.__set_estimated_stats(node.estimated_stats);
-        }
+        DCHECK(node.__isset.estimated_stats);
+        node_summary.__set_estimated_stats(node.estimated_stats);
         node_summary.exec_stats.resize(num_instances);
       }
 

http://git-wip-us.apache.org/repos/asf/impala/blob/d8792c21/be/src/runtime/coordinator.h
----------------------------------------------------------------------
diff --git a/be/src/runtime/coordinator.h b/be/src/runtime/coordinator.h
index 546efd5..3ab233a 100644
--- a/be/src/runtime/coordinator.h
+++ b/be/src/runtime/coordinator.h
@@ -281,6 +281,9 @@ class Coordinator { // NOLINT: The member variables could be re-ordered to save
     /// A mapping of plan node ids to index into thrift_exec_summary.nodes
     boost::unordered_map<TPlanNodeId, int> node_id_to_idx_map;
 
+    /// A mapping of fragment data sink ids to index into thrift_exec_summary.nodes
+    boost::unordered_map<TPlanNodeId, int> data_sink_id_to_idx_map;
+
     void Init(const QuerySchedule& query_schedule);
   };
 

http://git-wip-us.apache.org/repos/asf/impala/blob/d8792c21/be/src/runtime/data-stream-test.cc
----------------------------------------------------------------------
diff --git a/be/src/runtime/data-stream-test.cc b/be/src/runtime/data-stream-test.cc
index de5e349..c36e81c 100644
--- a/be/src/runtime/data-stream-test.cc
+++ b/be/src/runtime/data-stream-test.cc
@@ -535,7 +535,7 @@ class DataStreamTest : public testing::Test {
     TExpr output_exprs;
     output_exprs.nodes.push_back(expr_node);
 
-    sender.reset(new KrpcDataStreamSender(
+    sender.reset(new KrpcDataStreamSender(-1,
         sender_num, row_desc_, sink.stream_sink, dest_, channel_buffer_size, &state));
     EXPECT_OK(static_cast<KrpcDataStreamSender*>(
         sender.get())->Init(vector<TExpr>({output_exprs}), sink, &state));

http://git-wip-us.apache.org/repos/asf/impala/blob/d8792c21/be/src/runtime/krpc-data-stream-sender.cc
----------------------------------------------------------------------
diff --git a/be/src/runtime/krpc-data-stream-sender.cc b/be/src/runtime/krpc-data-stream-sender.cc
index acffcbe..6bf3b78 100644
--- a/be/src/runtime/krpc-data-stream-sender.cc
+++ b/be/src/runtime/krpc-data-stream-sender.cc
@@ -582,10 +582,11 @@ void KrpcDataStreamSender::Channel::Teardown(RuntimeState* state) {
   batch_.reset();
 }
 
-KrpcDataStreamSender::KrpcDataStreamSender(int sender_id, const RowDescriptor* row_desc,
-    const TDataStreamSink& sink, const vector<TPlanFragmentDestination>& destinations,
-    int per_channel_buffer_size, RuntimeState* state)
-  : DataSink(row_desc,
+KrpcDataStreamSender::KrpcDataStreamSender(TDataSinkId sink_id, int sender_id,
+    const RowDescriptor* row_desc, const TDataStreamSink& sink,
+    const vector<TPlanFragmentDestination>& destinations, int per_channel_buffer_size,
+    RuntimeState* state)
+  : DataSink(sink_id, row_desc,
         Substitute("KrpcDataStreamSender (dst_id=$0)", sink.dest_node_id), state),
     sender_id_(sender_id),
     partition_type_(sink.output_partition.type),

http://git-wip-us.apache.org/repos/asf/impala/blob/d8792c21/be/src/runtime/krpc-data-stream-sender.h
----------------------------------------------------------------------
diff --git a/be/src/runtime/krpc-data-stream-sender.h b/be/src/runtime/krpc-data-stream-sender.h
index 65580b1..bf3cfdc 100644
--- a/be/src/runtime/krpc-data-stream-sender.h
+++ b/be/src/runtime/krpc-data-stream-sender.h
@@ -60,7 +60,7 @@ class KrpcDataStreamSender : public DataSink {
   /// per-channel's accumulating row batch before it will be sent.
   /// NOTE: supported partition types are UNPARTITIONED (broadcast), HASH_PARTITIONED,
   /// and RANDOM.
-  KrpcDataStreamSender(int sender_id, const RowDescriptor* row_desc,
+  KrpcDataStreamSender(TDataSinkId sink_id, int sender_id, const RowDescriptor* row_desc,
       const TDataStreamSink& tsink,
       const std::vector<TPlanFragmentDestination>& destinations,
       int per_channel_buffer_size, RuntimeState* state);

http://git-wip-us.apache.org/repos/asf/impala/blob/d8792c21/be/src/util/runtime-profile.cc
----------------------------------------------------------------------
diff --git a/be/src/util/runtime-profile.cc b/be/src/util/runtime-profile.cc
index 9473cdf..44c32fe 100644
--- a/be/src/util/runtime-profile.cc
+++ b/be/src/util/runtime-profile.cc
@@ -63,11 +63,10 @@ RuntimeProfile* RuntimeProfile::Create(ObjectPool* pool, const string& name,
   return pool->Add(new RuntimeProfile(pool, name, is_averaged_profile));
 }
 
-RuntimeProfile::RuntimeProfile(ObjectPool* pool, const string& name,
-    bool is_averaged_profile)
+RuntimeProfile::RuntimeProfile(
+    ObjectPool* pool, const string& name, bool is_averaged_profile)
   : pool_(pool),
     name_(name),
-    metadata_(-1),
     is_averaged_profile_(is_averaged_profile),
     counter_total_time_(TUnit::TIME_NS),
     inactive_timer_(TUnit::TIME_NS),
@@ -123,11 +122,11 @@ RuntimeProfile* RuntimeProfile::CreateFromThrift(ObjectPool* pool,
 
   const TRuntimeProfileNode& node = nodes[*idx];
   RuntimeProfile* profile = Create(pool, node.name);
-  profile->metadata_ = node.metadata;
+  profile->metadata_ = node.node_metadata;
   for (int i = 0; i < node.counters.size(); ++i) {
     const TCounter& counter = node.counters[i];
     profile->counter_map_[counter.name] =
-      pool->Add(new Counter(counter.unit, counter.value));
+        pool->Add(new Counter(counter.unit, counter.value));
   }
 
   if (node.__isset.event_sequences) {
@@ -373,7 +372,7 @@ void RuntimeProfile::Update(const vector<TRuntimeProfileNode>& nodes, int* idx)
         }
       } else {
         child = Create(pool_, tchild.name);
-        child->metadata_ = tchild.metadata;
+        child->metadata_ = tchild.node_metadata;
         child_map_[tchild.name] = child;
         insert_pos = children_.insert(insert_pos, make_pair(child, tchild.indent));
         ++insert_pos;
@@ -913,7 +912,10 @@ void RuntimeProfile::ToThrift(vector<TRuntimeProfileNode>* nodes) const {
   nodes->push_back(TRuntimeProfileNode());
   TRuntimeProfileNode& node = (*nodes)[index];
   node.name = name_;
-  node.metadata = metadata_;
+  // Set the required metadata field to the plan node ID for compatibility with any tools
+  // that rely on the plan node id being set there.
+  node.metadata = metadata_.__isset.plan_node_id ? metadata_.plan_node_id : -1;
+  node.__set_node_metadata(metadata_);
   node.indent = true;
 
   CounterMap counter_map;
@@ -1004,11 +1006,19 @@ void RuntimeProfile::GetExecSummary(TExecSummary* t_exec_summary) const {
   *t_exec_summary = t_exec_summary_;
 }
 
+void RuntimeProfile::SetPlanNodeId(int node_id) {
+  DCHECK(!metadata_.__isset.data_sink_id) << "Don't set conflicting metadata";
+  metadata_.__set_plan_node_id(node_id);
+}
+
+void RuntimeProfile::SetDataSinkId(int sink_id) {
+  DCHECK(!metadata_.__isset.plan_node_id) << "Don't set conflicting metadata";
+  metadata_.__set_data_sink_id(sink_id);
+}
+
 int64_t RuntimeProfile::UnitsPerSecond(
-    const RuntimeProfile::Counter* total_counter,
-    const RuntimeProfile::Counter* timer) {
-  DCHECK(total_counter->unit() == TUnit::BYTES ||
-         total_counter->unit() == TUnit::UNIT);
+    const RuntimeProfile::Counter* total_counter, const RuntimeProfile::Counter* timer) {
+  DCHECK(total_counter->unit() == TUnit::BYTES || total_counter->unit() == TUnit::UNIT);
   DCHECK(timer->unit() == TUnit::TIME_NS);
 
   if (timer->value() == 0) return 0;

http://git-wip-us.apache.org/repos/asf/impala/blob/d8792c21/be/src/util/runtime-profile.h
----------------------------------------------------------------------
diff --git a/be/src/util/runtime-profile.h b/be/src/util/runtime-profile.h
index ce83361..c1c0462 100644
--- a/be/src/util/runtime-profile.h
+++ b/be/src/util/runtime-profile.h
@@ -305,12 +305,18 @@ class RuntimeProfile { // NOLINT: This struct is not packed, but there are not s
   /// (because it doesn't re-file child profiles)
   void set_name(const std::string& name) { name_ = name; }
 
-  int64_t metadata() const { return metadata_; }
-  void set_metadata(int64_t md) { metadata_ = md; }
+  const TRuntimeProfileNodeMetadata& metadata() const { return metadata_; }
+
+  /// Called if this corresponds to a plan node. Sets metadata so that later code that
+  /// analyzes the profile can identify this as the plan node's profile.
+  void SetPlanNodeId(int node_id);
+
+  /// Called if this corresponds to a data sink. Sets metadata so that later code that
+  /// analyzes the profile can identify this as the data sink's profile.
+  void SetDataSinkId(int sink_id);
 
   /// Derived counter function: return measured throughput as input_value/second.
-  static int64_t UnitsPerSecond(
-      const Counter* total_counter, const Counter* timer);
+  static int64_t UnitsPerSecond(const Counter* total_counter, const Counter* timer);
 
   /// Derived counter function: return aggregated value
   static int64_t CounterSum(const std::vector<Counter*>* counters);
@@ -387,8 +393,8 @@ class RuntimeProfile { // NOLINT: This struct is not packed, but there are not s
   /// Name for this runtime profile.
   std::string name_;
 
-  /// user-supplied, uninterpreted metadata.
-  int64_t metadata_;
+  /// Detailed metadata that identifies the plan node, sink, etc.
+  TRuntimeProfileNodeMetadata metadata_;
 
   /// True if this profile is an average derived from other profiles.
   /// All counters in this profile must be of unit AveragedCounter.

http://git-wip-us.apache.org/repos/asf/impala/blob/d8792c21/be/src/util/summary-util.cc
----------------------------------------------------------------------
diff --git a/be/src/util/summary-util.cc b/be/src/util/summary-util.cc
index 01b3d7e..ab90e87 100644
--- a/be/src/util/summary-util.cc
+++ b/be/src/util/summary-util.cc
@@ -76,10 +76,15 @@ void PrintExecSummary(const TExecSummary& exec_summary, int indent_level,
   row.push_back(lexical_cast<string>(node.exec_stats.size())); // Num instances
   row.push_back(PrettyPrinter::Print(avg_time, TUnit::TIME_NS));
   row.push_back(PrettyPrinter::Print(max_stats.latency_ns, TUnit::TIME_NS));
-  row.push_back(PrettyPrinter::Print(
-      node.is_broadcast ? max_stats.cardinality : agg_stats.cardinality,
-      TUnit::UNIT));
-  row.push_back(PrettyPrinter::Print(est_stats.cardinality, TUnit::UNIT));
+  if (node.node_id == -1) {
+    // Cardinality stats are not valid for sinks.
+    row.push_back("");
+    row.push_back("");
+  } else {
+    row.push_back(PrettyPrinter::Print(
+        node.is_broadcast ? max_stats.cardinality : agg_stats.cardinality, TUnit::UNIT));
+    row.push_back(PrettyPrinter::Print(est_stats.cardinality, TUnit::UNIT));
+  }
   row.push_back(PrettyPrinter::Print(max_stats.memory_used, TUnit::BYTES));
   row.push_back(PrettyPrinter::Print(est_stats.memory_used, TUnit::BYTES));
   // Node "details" may contain exprs which should be redacted.

http://git-wip-us.apache.org/repos/asf/impala/blob/d8792c21/common/thrift/DataSinks.thrift
----------------------------------------------------------------------
diff --git a/common/thrift/DataSinks.thrift b/common/thrift/DataSinks.thrift
index 1b26c92..d63b8a0 100644
--- a/common/thrift/DataSinks.thrift
+++ b/common/thrift/DataSinks.thrift
@@ -18,6 +18,7 @@
 namespace cpp impala
 namespace java org.apache.impala.thrift
 
+include "ExecStats.thrift"
 include "Exprs.thrift"
 include "Types.thrift"
 include "Descriptors.thrift"
@@ -106,8 +107,15 @@ struct TTableSink {
 }
 
 struct TDataSink {
+  // Tagged union of the different types of data sink.
   1: required TDataSinkType type
   2: optional TDataStreamSink stream_sink
   3: optional TTableSink table_sink
   4: optional TJoinBuildSink join_build_sink
+
+  // A human-readable label for the sink.
+  5: optional string label
+
+  // Estimated execution stats generated by the planner.
+  6: optional ExecStats.TExecStats estimated_stats
 }

http://git-wip-us.apache.org/repos/asf/impala/blob/d8792c21/common/thrift/ExecStats.thrift
----------------------------------------------------------------------
diff --git a/common/thrift/ExecStats.thrift b/common/thrift/ExecStats.thrift
index 9c759e9..94d5c9b 100644
--- a/common/thrift/ExecStats.thrift
+++ b/common/thrift/ExecStats.thrift
@@ -53,9 +53,10 @@ struct TExecStats {
   4: optional i64 memory_used
 }
 
-// Summary for a single plan node. This includes labels for how to display the
-// node as well as per instance stats.
+// Summary for a single plan node or data sink. This includes labels for how to display
+// the node as well as per instance stats.
 struct TPlanNodeExecSummary {
+  // The plan node ID or -1 if this is a data sink at the root of a fragment.
   1: required Types.TPlanNodeId node_id
   2: required Types.TFragmentIdx fragment_idx
   3: required string label
@@ -65,10 +66,10 @@ struct TPlanNodeExecSummary {
   // Estimated stats generated by the planner
   6: optional TExecStats estimated_stats
 
-  // One entry for each fragment instance executing this plan node.
+  // One entry for each fragment instance executing this plan node or data sink.
   7: optional list<TExecStats> exec_stats
 
-  // If true, this plan node is an exchange node that is the receiver of a broadcast.
+  // If true, this is an exchange node that is the receiver of a broadcast.
   8: optional bool is_broadcast
 }
 

http://git-wip-us.apache.org/repos/asf/impala/blob/d8792c21/common/thrift/RuntimeProfile.thrift
----------------------------------------------------------------------
diff --git a/common/thrift/RuntimeProfile.thrift b/common/thrift/RuntimeProfile.thrift
index b2396b7..f131564 100644
--- a/common/thrift/RuntimeProfile.thrift
+++ b/common/thrift/RuntimeProfile.thrift
@@ -20,6 +20,7 @@ namespace java org.apache.impala.thrift
 
 include "ExecStats.thrift"
 include "Metrics.thrift"
+include "Types.thrift"
 
 // Represents the different formats a runtime profile can be represented in.
 enum TRuntimeProfileFormat {
@@ -76,13 +77,23 @@ struct TSummaryStatsCounter {
   6: required i64 max_value
 }
 
+// Metadata to help identify what entity the profile node corresponds to.
+union TRuntimeProfileNodeMetadata {
+  // Set if this node corresponds to a plan node.
+  1: Types.TPlanNodeId plan_node_id
+
+  // Set if this node corresponds to a data sink.
+  2: Types.TDataSinkId data_sink_id
+}
+
 // A single runtime profile
 struct TRuntimeProfileNode {
   1: required string name
   2: required i32 num_children
   3: required list<TCounter> counters
-  // TODO: should we make metadata a serializable struct?  We only use it to
-  // store the node id right now so this is sufficient.
+
+  // Legacy field. May contain the node ID for plan nodes.
+  // Replaced by node_metadata, which contains richer metadata.
   4: required i64 metadata
 
   // indicates whether the child will be printed with extra indentation;
@@ -107,6 +118,9 @@ struct TRuntimeProfileNode {
 
   // List of summary stats counters
   11: optional list<TSummaryStatsCounter> summary_stats_counters
+
+  // Metadata about the entity that this node refers to.
+  12: optional TRuntimeProfileNodeMetadata node_metadata
 }
 
 // A flattened tree of runtime profiles, obtained by an

http://git-wip-us.apache.org/repos/asf/impala/blob/d8792c21/common/thrift/Types.thrift
----------------------------------------------------------------------
diff --git a/common/thrift/Types.thrift b/common/thrift/Types.thrift
index 6236f10..0d304ab 100644
--- a/common/thrift/Types.thrift
+++ b/common/thrift/Types.thrift
@@ -21,6 +21,7 @@ namespace java org.apache.impala.thrift
 typedef i64 TTimestamp
 typedef i32 TFragmentIdx
 typedef i32 TPlanNodeId
+typedef i32 TDataSinkId
 typedef i32 TTupleId
 typedef i32 TSlotId
 typedef i32 TTableId

http://git-wip-us.apache.org/repos/asf/impala/blob/d8792c21/fe/src/main/java/org/apache/impala/planner/DataSink.java
----------------------------------------------------------------------
diff --git a/fe/src/main/java/org/apache/impala/planner/DataSink.java b/fe/src/main/java/org/apache/impala/planner/DataSink.java
index 6acb4b9..cb4426c 100644
--- a/fe/src/main/java/org/apache/impala/planner/DataSink.java
+++ b/fe/src/main/java/org/apache/impala/planner/DataSink.java
@@ -18,6 +18,8 @@
 package org.apache.impala.planner;
 
 import org.apache.impala.thrift.TDataSink;
+import org.apache.impala.thrift.TDataSinkType;
+import org.apache.impala.thrift.TExecStats;
 import org.apache.impala.thrift.TExplainLevel;
 import org.apache.impala.thrift.TQueryOptions;
 
@@ -58,7 +60,33 @@ public abstract class DataSink {
   abstract protected void appendSinkExplainString(String prefix, String detailPrefix,
       TQueryOptions queryOptions, TExplainLevel explainLevel, StringBuilder output);
 
-  protected abstract TDataSink toThrift();
+  /**
+   * Return a short human-readable name to describe the sink in the exec summary.
+   */
+  abstract protected String getLabel();
+
+  /**
+   * Construct a thrift representation of the sink.
+   */
+  protected final TDataSink toThrift() {
+    TDataSink tsink = new TDataSink(getSinkType());
+    tsink.setLabel(fragment_.getId() + ":" + getLabel());
+    TExecStats estimatedStats = new TExecStats();
+    estimatedStats.setMemory_used(resourceProfile_.getMemEstimateBytes());
+    tsink.setEstimated_stats(estimatedStats);
+    toThriftImpl(tsink);
+    return tsink;
+  }
+
+  /**
+   * Add subclass-specific information to the sink.
+   */
+  abstract protected void toThriftImpl(TDataSink tsink);
+
+  /**
+   * Get the sink type of the subclass.
+   */
+  abstract protected TDataSinkType getSinkType();
 
   public void setFragment(PlanFragment fragment) { fragment_ = fragment; }
   public PlanFragment getFragment() { return fragment_; }

http://git-wip-us.apache.org/repos/asf/impala/blob/d8792c21/fe/src/main/java/org/apache/impala/planner/DataStreamSink.java
----------------------------------------------------------------------
diff --git a/fe/src/main/java/org/apache/impala/planner/DataStreamSink.java b/fe/src/main/java/org/apache/impala/planner/DataStreamSink.java
index af4f9a6..5a2a8b4 100644
--- a/fe/src/main/java/org/apache/impala/planner/DataStreamSink.java
+++ b/fe/src/main/java/org/apache/impala/planner/DataStreamSink.java
@@ -50,17 +50,25 @@ public class DataStreamSink extends DataSink {
   }
 
   @Override
+  protected String getLabel() {
+    return "EXCHANGE SENDER";
+  }
+
+  @Override
   public void computeResourceProfile(TQueryOptions queryOptions) {
     resourceProfile_ = ResourceProfile.noReservation(0);
   }
 
   @Override
-  protected TDataSink toThrift() {
-    TDataSink result = new TDataSink(TDataSinkType.DATA_STREAM_SINK);
+  protected void toThriftImpl(TDataSink tsink) {
     TDataStreamSink tStreamSink =
         new TDataStreamSink(exchNode_.getId().asInt(), outputPartition_.toThrift());
-    result.setStream_sink(tStreamSink);
-    return result;
+    tsink.setStream_sink(tStreamSink);
+  }
+
+  @Override
+  protected TDataSinkType getSinkType() {
+    return TDataSinkType.DATA_STREAM_SINK;
   }
 
   public DataPartition getOutputPartition() { return outputPartition_; }

http://git-wip-us.apache.org/repos/asf/impala/blob/d8792c21/fe/src/main/java/org/apache/impala/planner/HBaseTableSink.java
----------------------------------------------------------------------
diff --git a/fe/src/main/java/org/apache/impala/planner/HBaseTableSink.java b/fe/src/main/java/org/apache/impala/planner/HBaseTableSink.java
index 75042e6..abdf2db 100644
--- a/fe/src/main/java/org/apache/impala/planner/HBaseTableSink.java
+++ b/fe/src/main/java/org/apache/impala/planner/HBaseTableSink.java
@@ -43,16 +43,25 @@ public class HBaseTableSink extends TableSink {
   }
 
   @Override
+  protected String getLabel() {
+    return "HBASE WRITER";
+  }
+
+  @Override
   public void computeResourceProfile(TQueryOptions queryOptions) {
     resourceProfile_ = ResourceProfile.noReservation(0);
   }
 
   @Override
-  protected TDataSink toThrift() {
-    TDataSink result = new TDataSink(TDataSinkType.TABLE_SINK);
+  protected void toThriftImpl(TDataSink tsink) {
     TTableSink tTableSink = new TTableSink(DescriptorTable.TABLE_SINK_ID,
         TTableSinkType.HBASE, sinkOp_.toThrift());
-    result.table_sink = tTableSink;
-    return result;
+    tsink.table_sink = tTableSink;
   }
+
+  @Override
+  protected TDataSinkType getSinkType() {
+    return TDataSinkType.TABLE_SINK;
+  }
+
 }

http://git-wip-us.apache.org/repos/asf/impala/blob/d8792c21/fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
----------------------------------------------------------------------
diff --git a/fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java b/fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
index cd99cff..12f4e67 100644
--- a/fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
+++ b/fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
@@ -174,8 +174,12 @@ public class HdfsTableSink extends TableSink {
   }
 
   @Override
-  protected TDataSink toThrift() {
-    TDataSink result = new TDataSink(TDataSinkType.TABLE_SINK);
+  protected String getLabel() {
+    return "HDFS WRITER";
+  }
+
+  @Override
+  protected void toThriftImpl(TDataSink tsink) {
     THdfsTableSink hdfsTableSink = new THdfsTableSink(
         Expr.treesToThrift(partitionKeyExprs_), overwrite_, inputIsClustered_);
     FeFsTable table = (FeFsTable) targetTable_;
@@ -190,7 +194,11 @@ public class HdfsTableSink extends TableSink {
     TTableSink tTableSink = new TTableSink(DescriptorTable.TABLE_SINK_ID,
         TTableSinkType.HDFS, sinkOp_.toThrift());
     tTableSink.hdfs_table_sink = hdfsTableSink;
-    result.table_sink = tTableSink;
-    return result;
+    tsink.table_sink = tTableSink;
+  }
+
+  @Override
+  protected TDataSinkType getSinkType() {
+    return TDataSinkType.TABLE_SINK;
   }
 }

http://git-wip-us.apache.org/repos/asf/impala/blob/d8792c21/fe/src/main/java/org/apache/impala/planner/JoinBuildSink.java
----------------------------------------------------------------------
diff --git a/fe/src/main/java/org/apache/impala/planner/JoinBuildSink.java b/fe/src/main/java/org/apache/impala/planner/JoinBuildSink.java
index a9bea59..b37c326 100644
--- a/fe/src/main/java/org/apache/impala/planner/JoinBuildSink.java
+++ b/fe/src/main/java/org/apache/impala/planner/JoinBuildSink.java
@@ -59,15 +59,18 @@ public class JoinBuildSink extends DataSink {
   public JoinTableId getJoinTableId() { return joinTableId_; }
 
   @Override
-  protected TDataSink toThrift() {
-    TDataSink result = new TDataSink(TDataSinkType.JOIN_BUILD_SINK);
+  protected void toThriftImpl(TDataSink tsink) {
     TJoinBuildSink tBuildSink = new TJoinBuildSink();
     tBuildSink.setJoin_table_id(joinTableId_.asInt());
     for (Expr buildExpr: buildExprs_) {
       tBuildSink.addToBuild_exprs(buildExpr.treeToThrift());
     }
-    result.setJoin_build_sink(tBuildSink);
-    return result;
+    tsink.setJoin_build_sink(tBuildSink);
+  }
+
+  @Override
+  protected TDataSinkType getSinkType() {
+    return TDataSinkType.JOIN_BUILD_SINK;
   }
 
   @Override
@@ -87,6 +90,11 @@ public class JoinBuildSink extends DataSink {
   }
 
   @Override
+  protected String getLabel() {
+    return "JOIN BUILD";
+  }
+
+  @Override
   public void computeResourceProfile(TQueryOptions queryOptions) {
     // The memory consumption is counted against the join PlanNode.
     resourceProfile_ = ResourceProfile.noReservation(0);

http://git-wip-us.apache.org/repos/asf/impala/blob/d8792c21/fe/src/main/java/org/apache/impala/planner/KuduTableSink.java
----------------------------------------------------------------------
diff --git a/fe/src/main/java/org/apache/impala/planner/KuduTableSink.java b/fe/src/main/java/org/apache/impala/planner/KuduTableSink.java
index ef7cbab..bb57b1f 100644
--- a/fe/src/main/java/org/apache/impala/planner/KuduTableSink.java
+++ b/fe/src/main/java/org/apache/impala/planner/KuduTableSink.java
@@ -58,20 +58,28 @@ public class KuduTableSink extends TableSink {
   }
 
   @Override
+  protected String getLabel() {
+    return "KUDU WRITER";
+  }
+
+  @Override
   public void computeResourceProfile(TQueryOptions queryOptions) {
     // TODO: add a memory estimate
     resourceProfile_ = ResourceProfile.noReservation(0);
   }
 
   @Override
-  protected TDataSink toThrift() {
-    TDataSink result = new TDataSink(TDataSinkType.TABLE_SINK);
+  protected void toThriftImpl(TDataSink tsink) {
     TTableSink tTableSink = new TTableSink(DescriptorTable.TABLE_SINK_ID,
         TTableSinkType.KUDU, sinkOp_.toThrift());
     TKuduTableSink tKuduSink = new TKuduTableSink();
     tKuduSink.setReferenced_columns(targetColIdxs_);
     tTableSink.setKudu_table_sink(tKuduSink);
-    result.table_sink = tTableSink;
-    return result;
+    tsink.table_sink = tTableSink;
+  }
+
+  @Override
+  protected TDataSinkType getSinkType() {
+    return TDataSinkType.TABLE_SINK;
   }
 }

http://git-wip-us.apache.org/repos/asf/impala/blob/d8792c21/fe/src/main/java/org/apache/impala/planner/PlanRootSink.java
----------------------------------------------------------------------
diff --git a/fe/src/main/java/org/apache/impala/planner/PlanRootSink.java b/fe/src/main/java/org/apache/impala/planner/PlanRootSink.java
index 07eb58b..5a81bec 100644
--- a/fe/src/main/java/org/apache/impala/planner/PlanRootSink.java
+++ b/fe/src/main/java/org/apache/impala/planner/PlanRootSink.java
@@ -35,12 +35,21 @@ public class PlanRootSink extends DataSink {
   }
 
   @Override
+  protected String getLabel() {
+    return "ROOT";
+  }
+
+  @Override
   public void computeResourceProfile(TQueryOptions queryOptions) {
     // TODO: add a memory estimate
     resourceProfile_ = ResourceProfile.noReservation(0);
   }
 
-  protected TDataSink toThrift() {
-    return new TDataSink(TDataSinkType.PLAN_ROOT_SINK);
+  protected void toThriftImpl(TDataSink tsink) {
+  }
+
+  @Override
+  protected TDataSinkType getSinkType() {
+    return TDataSinkType.PLAN_ROOT_SINK;
   }
 }

http://git-wip-us.apache.org/repos/asf/impala/blob/d8792c21/shell/impala_client.py
----------------------------------------------------------------------
diff --git a/shell/impala_client.py b/shell/impala_client.py
index 3690fde..02f339a 100755
--- a/shell/impala_client.py
+++ b/shell/impala_client.py
@@ -203,12 +203,13 @@ class ImpalaClient(object):
     hosts = 0
     if node.exec_stats is not None:
       hosts = len(node.exec_stats)
+    is_sink = node.node_id == -1
     row = [ label_prefix + node.label,
             hosts,
             prettyprint_time(avg_time),
             prettyprint_time(max_stats.latency_ns),
-            prettyprint_units(cardinality),
-            prettyprint_units(est_stats.cardinality),
+            "" if is_sink else prettyprint_units(cardinality),
+            "" if is_sink else prettyprint_units(est_stats.cardinality),
             prettyprint_bytes(max_stats.memory_used),
             prettyprint_bytes(est_stats.memory_used),
             node.label_detail ]

http://git-wip-us.apache.org/repos/asf/impala/blob/d8792c21/tests/beeswax/impala_beeswax.py
----------------------------------------------------------------------
diff --git a/tests/beeswax/impala_beeswax.py b/tests/beeswax/impala_beeswax.py
index d8272ed..4056011 100644
--- a/tests/beeswax/impala_beeswax.py
+++ b/tests/beeswax/impala_beeswax.py
@@ -282,11 +282,14 @@ class ImpalaBeeswaxClient(object):
       row["num_hosts"] = len(node.exec_stats)
       row["avg_time"] = avg_time
 
+    is_sink = node.node_id == -1
     # If the node is a broadcast-receiving exchange node, the cardinality of rows produced
     # is the max over all instances (which should all have received the same number of
     # rows). Otherwise, the cardinality is the sum over all instances which process
     # disjoint partitions.
-    if node.is_broadcast:
+    if is_sink:
+      cardinality = -1
+    elif node.is_broadcast:
       cardinality = max_stats.cardinality
     else:
       cardinality = agg_stats.cardinality

http://git-wip-us.apache.org/repos/asf/impala/blob/d8792c21/tests/query_test/test_observability.py
----------------------------------------------------------------------
diff --git a/tests/query_test/test_observability.py b/tests/query_test/test_observability.py
index dd955d4..7f4b2c8 100644
--- a/tests/query_test/test_observability.py
+++ b/tests/query_test/test_observability.py
@@ -40,10 +40,11 @@ class TestObservability(ImpalaTestSuite):
     query = """select tinyint_col, count(*) from functional.alltypes
         group by tinyint_col order by tinyint_col limit 5"""
     result = self.execute_query(query)
-    assert result.exec_summary[0]['operator'] == '05:MERGING-EXCHANGE'
-    assert result.exec_summary[0]['num_rows'] == 5
-    assert result.exec_summary[0]['est_num_rows'] == 5
-    assert result.exec_summary[0]['peak_mem'] > 0
+    exchange = result.exec_summary[1]
+    assert exchange['operator'] == '05:MERGING-EXCHANGE'
+    assert exchange['num_rows'] == 5
+    assert exchange['est_num_rows'] == 5
+    assert exchange['peak_mem'] > 0
 
     for line in result.runtime_profile.split('\n'):
       # The first 'RowsProduced' we find is for the coordinator fragment.
@@ -58,10 +59,11 @@ class TestObservability(ImpalaTestSuite):
         inner join functional.alltypessmall b on (a.id = b.id)
         where a.year = 2009 and b.month = 2"""
     result = self.execute_query(query)
-    assert result.exec_summary[5]['operator'] == '04:EXCHANGE'
-    assert result.exec_summary[5]['num_rows'] == 25
-    assert result.exec_summary[5]['est_num_rows'] == 25
-    assert result.exec_summary[5]['peak_mem'] > 0
+    exchange = result.exec_summary[8]
+    assert exchange['operator'] == '04:EXCHANGE'
+    assert exchange['num_rows'] == 25
+    assert exchange['est_num_rows'] == 25
+    assert exchange['peak_mem'] > 0
 
   def test_report_time(self):
     """ Regression test for IMPALA-6741 - checks that last reporting time exists in
@@ -116,6 +118,45 @@ class TestObservability(ImpalaTestSuite):
     assert result.exec_summary[scan_idx]['operator'] == '00:SCAN HBASE'
     assert result.exec_summary[scan_idx]['detail'] == 'functional_hbase.alltypestiny'
 
+  def test_sink_summary(self, unique_database):
+    """IMPALA-1048: Checks that the exec summary contains sinks."""
+    # SELECT query.
+    query = "select count(*) from functional.alltypes"
+    result = self.execute_query(query)
+    # Sanity-check the root sink.
+    root_sink = result.exec_summary[0]
+    assert root_sink['operator'] == 'F01:ROOT'
+    assert root_sink['max_time'] >= 0
+    assert root_sink['num_rows'] == -1
+    assert root_sink['est_num_rows'] == -1
+    assert root_sink['peak_mem'] >= 0
+    assert root_sink['est_peak_mem'] >= 0
+    # Sanity-check the exchange sink.
+    found_exchange_sender = False
+    for row in result.exec_summary[1:]:
+      if 'EXCHANGE SENDER' not in row['operator']:
+        continue
+      found_exchange_sender = True
+      assert re.match("F[0-9]+:EXCHANGE SENDER", row['operator'])
+      assert row['max_time'] >= 0
+      assert row['num_rows'] == -1
+      assert row['est_num_rows'] == -1
+      assert row['peak_mem'] >= 0
+      assert row['est_peak_mem'] >= 0
+    assert found_exchange_sender, result
+
+    # INSERT query.
+    query = "create table {0}.tmp as select count(*) from functional.alltypes".format(
+        unique_database)
+    result = self.execute_query(query)
+    # Sanity-check the HDFS writer sink.
+    assert result.exec_summary[0]['operator'] == 'F01:HDFS WRITER'
+    assert result.exec_summary[0]['max_time'] >= 0
+    assert result.exec_summary[0]['num_rows'] == -1
+    assert result.exec_summary[0]['est_num_rows'] == -1
+    assert result.exec_summary[0]['peak_mem'] >= 0
+    assert result.exec_summary[0]['est_peak_mem'] >= 0
+
   def test_query_states(self):
     """Tests that the query profile shows expected query states."""
     query = "select count(*) from functional.alltypes"