You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2025/10/17 16:22:43 UTC
(datafusion) branch main updated: feat: Support configurable `EXPLAIN ANALYZE` detail level (#18098)
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion.git
The following commit(s) were added to refs/heads/main by this push:
new ec2402aee9 feat: Support configurable `EXPLAIN ANALYZE` detail level (#18098)
ec2402aee9 is described below
commit ec2402aee9bf510d3a98927cb3580850914fcf27
Author: Yongting You <20...@gmail.com>
AuthorDate: Sat Oct 18 00:22:21 2025 +0800
feat: Support configurable `EXPLAIN ANALYZE` detail level (#18098)
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes #123` indicates that this PR will close issue #123.
-->
- Closes #.
## Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->
`EXPLAIN ANALYZE` can be used for profiling and displays the results
alongside the EXPLAIN plan. The issue is that it currently shows too
many low-level details. It would provide a better user experience if
only the most commonly used metrics were shown by default, with more
detailed metrics available through specific configuration options.
### Example
In `datafusion-cli`:
```
> CREATE EXTERNAL TABLE IF NOT EXISTS lineitem
STORED AS parquet
LOCATION '/Users/yongting/Code/datafusion/benchmarks/data/tpch_sf1/lineitem';
0 row(s) fetched.
Elapsed 0.000 seconds.
explain analyze select *
from lineitem
where l_orderkey = 3000000;
```
The parquet reader includes a large number of low-level details:
```
metrics=[output_rows=19813, elapsed_compute=14ns, batches_split=0, bytes_scanned=2147308, file_open_errors=0, file_scan_errors=0, files_ranges_pruned_statistics=18, num_predicate_creation_errors=0, page_index_rows_matched=19813, page_index_rows_pruned=729088, predicate_cache_inner_records=0, predicate_cache_records=0, predicate_evaluation_errors=0, pushdown_rows_matched=0, pushdown_rows_pruned=0, row_groups_matched_bloom_filter=0, row_groups_matched_statistics=1, row_groups_pruned_blo [...]
```
I believe only a subset of it is commonly used, for example
`output_rows`, `metadata_load_time`, and how many file/row-group/pages
are pruned, and it would better to only display the most common ones by
default.
### Existing `VERBOSE` keyword
There is a existing verbose keyword in `EXPLAIN ANALYZE VERBOSE`,
however it's turning on per-partition metrics instead of controlling
detail level. I think it would be hard to mix this partition control and
the detail level introduced in this PR, so they're separated: the
following config will be used for detail level and the semantics of
`EXPLAIN ANALYZE VERBOSE` keep unchanged.
### This PR: configurable explain analyze level
1. Introduced a new config option `datafusion.explain.analyze_level`.
When set to `dev` (default value), all existing metrics will be shown.
If set to `summary`, only `BaselineMetrics` will be displayed (i.e.
`output_rows` and `elapsed_compute`).
Note now we only include `BaselineMetrics` for simplicity, in the
follow-up PRs we can figure out what's the commonly used metrics for
each operator, and add them to `summary` analyze level, finally set the
`summary` analyze level to default.
2. Add a `MetricType` field associated with `Metric` for detail level or
potentially category in the future. For different configurations, a
certain `MetricType` set will be shown accordingly.
#### Demo
```
-- continuing the above example
> set datafusion.explain.analyze_level = summary;
0 row(s) fetched.
Elapsed 0.000 seconds.
> explain analyze select *
from lineitem
where l_orderkey = 3000000;
+-------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ [...]
| plan_type | plan [...]
+-------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ [...]
| Plan with Metrics | CoalesceBatchesExec: target_batch_size=8192, metrics=[output_rows=5, elapsed_compute=25.339µs] [...]
| | FilterExec: l_orderkey@0 = 3000000, metrics=[output_rows=5, elapsed_compute=81.221µs] [...]
| | DataSourceExec: file_groups={14 groups: [[Users/yongting/Code/datafusion/benchmarks/data/tpch_sf1/lineitem/part-0.parquet:0..11525426], [Users/yongting/Code/datafusion/benchmarks/data/tpch_sf1/lineitem/part-0.parquet:11525426..20311205, Users/yongting/Code/datafusion/benchmarks/data/tpch_sf1/lineitem/part-1.parquet:0..2739647], [Users/yongting/Code/datafusion/benchmarks/data/tpch_sf1/lineitem/part-1.parquet:2739647..14265073], [Users/yongting/Code/datafusion/ [...]
| | [...]
+-------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ [...]
1 row(s) fetched.
Elapsed 0.025 seconds.
```
Only `BaselineMetrics` are shown.
## What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->
## Are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
4. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->
UT
## Are there any user-facing changes?
No
<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->
---------
Co-authored-by: Andrew Lamb <an...@nerdnetworks.org>
---
datafusion/common/src/config.rs | 7 ++-
datafusion/common/src/format.rs | 45 ++++++++++++++++++
.../core/src/datasource/physical_plan/parquet.rs | 5 +-
datafusion/core/src/physical_planner.rs | 8 ++++
datafusion/core/tests/sql/explain_analyze.rs | 35 ++++++++++++++
datafusion/physical-plan/src/analyze.rs | 19 +++++++-
datafusion/physical-plan/src/display.rs | 33 +++++++++++++
datafusion/physical-plan/src/metrics/baseline.rs | 12 +++--
datafusion/physical-plan/src/metrics/builder.rs | 21 ++++++++-
datafusion/physical-plan/src/metrics/mod.rs | 54 +++++++++++++++++++++-
datafusion/proto/src/physical_plan/mod.rs | 2 +
.../proto/tests/cases/roundtrip_physical_plan.rs | 2 +
.../sqllogictest/test_files/information_schema.slt | 2 +
.../test_files/spark/aggregate/avg.slt | 2 +-
.../test_files/spark/string/concat.slt | 2 +-
.../test_files/spark/string/format_string.slt | 8 ++--
docs/source/user-guide/configs.md | 1 +
docs/source/user-guide/sql/explain.md | 11 +++--
18 files changed, 251 insertions(+), 18 deletions(-)
diff --git a/datafusion/common/src/config.rs b/datafusion/common/src/config.rs
index 126935a1de..52e3598569 100644
--- a/datafusion/common/src/config.rs
+++ b/datafusion/common/src/config.rs
@@ -22,7 +22,7 @@ use arrow_ipc::CompressionType;
#[cfg(feature = "parquet_encryption")]
use crate::encryption::{FileDecryptionProperties, FileEncryptionProperties};
use crate::error::_config_err;
-use crate::format::ExplainFormat;
+use crate::format::{ExplainAnalyzeLevel, ExplainFormat};
use crate::parsers::CompressionTypeVariant;
use crate::utils::get_available_parallelism;
use crate::{DataFusionError, Result};
@@ -991,6 +991,11 @@ config_namespace! {
/// (format=tree only) Maximum total width of the rendered tree.
/// When set to 0, the tree will have no width limit.
pub tree_maximum_render_width: usize, default = 240
+
+ /// Verbosity level for "EXPLAIN ANALYZE". Default is "dev"
+ /// "summary" shows common metrics for high-level insights.
+ /// "dev" provides deep operator-level introspection for developers.
+ pub analyze_level: ExplainAnalyzeLevel, default = ExplainAnalyzeLevel::Dev
}
}
diff --git a/datafusion/common/src/format.rs b/datafusion/common/src/format.rs
index 06ec519ef3..764190e118 100644
--- a/datafusion/common/src/format.rs
+++ b/datafusion/common/src/format.rs
@@ -205,3 +205,48 @@ impl ConfigField for ExplainFormat {
Ok(())
}
}
+
+/// Verbosity levels controlling how `EXPLAIN ANALYZE` renders metrics
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+pub enum ExplainAnalyzeLevel {
+ /// Show a compact view containing high-level metrics
+ Summary,
+ /// Show a developer-focused view with per-operator details
+ Dev,
+ // When adding new enum, update the error message in `from_str()` accordingly.
+}
+
+impl FromStr for ExplainAnalyzeLevel {
+ type Err = DataFusionError;
+
+ fn from_str(level: &str) -> Result<Self, Self::Err> {
+ match level.to_lowercase().as_str() {
+ "summary" => Ok(ExplainAnalyzeLevel::Summary),
+ "dev" => Ok(ExplainAnalyzeLevel::Dev),
+ other => Err(DataFusionError::Configuration(format!(
+ "Invalid explain analyze level. Expected 'summary' or 'dev'. Got '{other}'"
+ ))),
+ }
+ }
+}
+
+impl Display for ExplainAnalyzeLevel {
+ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+ let s = match self {
+ ExplainAnalyzeLevel::Summary => "summary",
+ ExplainAnalyzeLevel::Dev => "dev",
+ };
+ write!(f, "{s}")
+ }
+}
+
+impl ConfigField for ExplainAnalyzeLevel {
+ fn visit<V: Visit>(&self, v: &mut V, key: &str, description: &'static str) {
+ v.some(key, self, description)
+ }
+
+ fn set(&mut self, _: &str, value: &str) -> Result<()> {
+ *self = ExplainAnalyzeLevel::from_str(value)?;
+ Ok(())
+ }
+}
diff --git a/datafusion/core/src/datasource/physical_plan/parquet.rs b/datafusion/core/src/datasource/physical_plan/parquet.rs
index d0774e5717..10a475c1cc 100644
--- a/datafusion/core/src/datasource/physical_plan/parquet.rs
+++ b/datafusion/core/src/datasource/physical_plan/parquet.rs
@@ -64,7 +64,9 @@ mod tests {
use datafusion_physical_expr::planner::logical2physical;
use datafusion_physical_plan::analyze::AnalyzeExec;
use datafusion_physical_plan::collect;
- use datafusion_physical_plan::metrics::{ExecutionPlanMetricsSet, MetricsSet};
+ use datafusion_physical_plan::metrics::{
+ ExecutionPlanMetricsSet, MetricType, MetricsSet,
+ };
use datafusion_physical_plan::{ExecutionPlan, ExecutionPlanProperties};
use chrono::{TimeZone, Utc};
@@ -238,6 +240,7 @@ mod tests {
let analyze_exec = Arc::new(AnalyzeExec::new(
false,
false,
+ vec![MetricType::SUMMARY, MetricType::DEV],
// use a new ParquetSource to avoid sharing execution metrics
self.build_parquet_exec(
Arc::clone(table_schema),
diff --git a/datafusion/core/src/physical_planner.rs b/datafusion/core/src/physical_planner.rs
index c28e56790e..0fa17deea1 100644
--- a/datafusion/core/src/physical_planner.rs
+++ b/datafusion/core/src/physical_planner.rs
@@ -62,6 +62,7 @@ use arrow::compute::SortOptions;
use arrow::datatypes::Schema;
use datafusion_catalog::ScanArgs;
use datafusion_common::display::ToStringifiedPlan;
+use datafusion_common::format::ExplainAnalyzeLevel;
use datafusion_common::tree_node::{TreeNode, TreeNodeRecursion, TreeNodeVisitor};
use datafusion_common::TableReference;
use datafusion_common::{
@@ -90,6 +91,7 @@ use datafusion_physical_expr::{
use datafusion_physical_optimizer::PhysicalOptimizerRule;
use datafusion_physical_plan::empty::EmptyExec;
use datafusion_physical_plan::execution_plan::InvariantLevel;
+use datafusion_physical_plan::metrics::MetricType;
use datafusion_physical_plan::placeholder_row::PlaceholderRowExec;
use datafusion_physical_plan::recursive_query::RecursiveQueryExec;
use datafusion_physical_plan::unnest::ListUnnest;
@@ -2073,9 +2075,15 @@ impl DefaultPhysicalPlanner {
let input = self.create_physical_plan(&a.input, session_state).await?;
let schema = Arc::clone(a.schema.inner());
let show_statistics = session_state.config_options().explain.show_statistics;
+ let analyze_level = session_state.config_options().explain.analyze_level;
+ let metric_types = match analyze_level {
+ ExplainAnalyzeLevel::Summary => vec![MetricType::SUMMARY],
+ ExplainAnalyzeLevel::Dev => vec![MetricType::SUMMARY, MetricType::DEV],
+ };
Ok(Arc::new(AnalyzeExec::new(
a.verbose,
show_statistics,
+ metric_types,
input,
schema,
)))
diff --git a/datafusion/core/tests/sql/explain_analyze.rs b/datafusion/core/tests/sql/explain_analyze.rs
index e082cabaad..54a57ed901 100644
--- a/datafusion/core/tests/sql/explain_analyze.rs
+++ b/datafusion/core/tests/sql/explain_analyze.rs
@@ -22,6 +22,7 @@ use rstest::rstest;
use datafusion::config::ConfigOptions;
use datafusion::physical_plan::display::DisplayableExecutionPlan;
use datafusion::physical_plan::metrics::Timestamp;
+use datafusion_common::format::ExplainAnalyzeLevel;
use object_store::path::Path;
#[tokio::test]
@@ -158,6 +159,40 @@ async fn explain_analyze_baseline_metrics() {
fn nanos_from_timestamp(ts: &Timestamp) -> i64 {
ts.value().unwrap().timestamp_nanos_opt().unwrap()
}
+
+// Test different detail level for config `datafusion.explain.analyze_level`
+#[tokio::test]
+async fn explain_analyze_level() {
+ async fn collect_plan(level: ExplainAnalyzeLevel) -> String {
+ let mut config = SessionConfig::new();
+ config.options_mut().explain.analyze_level = level;
+ let ctx = SessionContext::new_with_config(config);
+ let sql = "EXPLAIN ANALYZE \
+ SELECT * \
+ FROM generate_series(10) as t1(v1) \
+ ORDER BY v1 DESC";
+ let dataframe = ctx.sql(sql).await.unwrap();
+ let batches = dataframe.collect().await.unwrap();
+ arrow::util::pretty::pretty_format_batches(&batches)
+ .unwrap()
+ .to_string()
+ }
+
+ for (level, needle, should_contain) in [
+ (ExplainAnalyzeLevel::Summary, "spill_count", false),
+ (ExplainAnalyzeLevel::Summary, "output_rows", true),
+ (ExplainAnalyzeLevel::Dev, "spill_count", true),
+ (ExplainAnalyzeLevel::Dev, "output_rows", true),
+ ] {
+ let plan = collect_plan(level).await;
+ assert_eq!(
+ plan.contains(needle),
+ should_contain,
+ "plan for level {level:?} unexpected content: {plan}"
+ );
+ }
+}
+
#[tokio::test]
async fn csv_explain_plans() {
// This test verify the look of each plan in its full cycle plan creation
diff --git a/datafusion/physical-plan/src/analyze.rs b/datafusion/physical-plan/src/analyze.rs
index c095afe5e7..c696cf5aa5 100644
--- a/datafusion/physical-plan/src/analyze.rs
+++ b/datafusion/physical-plan/src/analyze.rs
@@ -26,6 +26,7 @@ use super::{
SendableRecordBatchStream,
};
use crate::display::DisplayableExecutionPlan;
+use crate::metrics::MetricType;
use crate::{DisplayFormatType, ExecutionPlan, Partitioning};
use arrow::{array::StringBuilder, datatypes::SchemaRef, record_batch::RecordBatch};
@@ -44,6 +45,8 @@ pub struct AnalyzeExec {
verbose: bool,
/// If statistics should be displayed
show_statistics: bool,
+ /// Which metric categories should be displayed
+ metric_types: Vec<MetricType>,
/// The input plan (the plan being analyzed)
pub(crate) input: Arc<dyn ExecutionPlan>,
/// The output schema for RecordBatches of this exec node
@@ -56,6 +59,7 @@ impl AnalyzeExec {
pub fn new(
verbose: bool,
show_statistics: bool,
+ metric_types: Vec<MetricType>,
input: Arc<dyn ExecutionPlan>,
schema: SchemaRef,
) -> Self {
@@ -63,6 +67,7 @@ impl AnalyzeExec {
AnalyzeExec {
verbose,
show_statistics,
+ metric_types,
input,
schema,
cache,
@@ -145,6 +150,7 @@ impl ExecutionPlan for AnalyzeExec {
Ok(Arc::new(Self::new(
self.verbose,
self.show_statistics,
+ self.metric_types.clone(),
children.pop().unwrap(),
Arc::clone(&self.schema),
)))
@@ -182,6 +188,7 @@ impl ExecutionPlan for AnalyzeExec {
let captured_schema = Arc::clone(&self.schema);
let verbose = self.verbose;
let show_statistics = self.show_statistics;
+ let metric_types = self.metric_types.clone();
// future that gathers the results from all the tasks in the
// JoinSet that computes the overall row count and final
@@ -201,6 +208,7 @@ impl ExecutionPlan for AnalyzeExec {
duration,
captured_input,
captured_schema,
+ &metric_types,
)
};
@@ -219,6 +227,7 @@ fn create_output_batch(
duration: std::time::Duration,
input: Arc<dyn ExecutionPlan>,
schema: SchemaRef,
+ metric_types: &[MetricType],
) -> Result<RecordBatch> {
let mut type_builder = StringBuilder::with_capacity(1, 1024);
let mut plan_builder = StringBuilder::with_capacity(1, 1024);
@@ -227,6 +236,7 @@ fn create_output_batch(
type_builder.append_value("Plan with Metrics");
let annotated_plan = DisplayableExecutionPlan::with_metrics(input.as_ref())
+ .set_metric_types(metric_types.to_vec())
.set_show_statistics(show_statistics)
.indent(verbose)
.to_string();
@@ -238,6 +248,7 @@ fn create_output_batch(
type_builder.append_value("Plan with Full Metrics");
let annotated_plan = DisplayableExecutionPlan::with_full_metrics(input.as_ref())
+ .set_metric_types(metric_types.to_vec())
.set_show_statistics(show_statistics)
.indent(verbose)
.to_string();
@@ -282,7 +293,13 @@ mod tests {
let blocking_exec = Arc::new(BlockingExec::new(Arc::clone(&schema), 1));
let refs = blocking_exec.refs();
- let analyze_exec = Arc::new(AnalyzeExec::new(true, false, blocking_exec, schema));
+ let analyze_exec = Arc::new(AnalyzeExec::new(
+ true,
+ false,
+ vec![MetricType::SUMMARY, MetricType::DEV],
+ blocking_exec,
+ schema,
+ ));
let fut = collect(analyze_exec, task_ctx);
let mut fut = fut.boxed();
diff --git a/datafusion/physical-plan/src/display.rs b/datafusion/physical-plan/src/display.rs
index 2420edfc74..35ca0b65ae 100644
--- a/datafusion/physical-plan/src/display.rs
+++ b/datafusion/physical-plan/src/display.rs
@@ -28,6 +28,7 @@ use datafusion_common::display::{GraphvizBuilder, PlanType, StringifiedPlan};
use datafusion_expr::display_schema;
use datafusion_physical_expr::LexOrdering;
+use crate::metrics::MetricType;
use crate::render_tree::RenderTree;
use super::{accept, ExecutionPlan, ExecutionPlanVisitor};
@@ -120,11 +121,17 @@ pub struct DisplayableExecutionPlan<'a> {
show_statistics: bool,
/// If schema should be displayed. See [`Self::set_show_schema`]
show_schema: bool,
+ /// Which metric categories should be included when rendering
+ metric_types: Vec<MetricType>,
// (TreeRender) Maximum total width of the rendered tree
tree_maximum_render_width: usize,
}
impl<'a> DisplayableExecutionPlan<'a> {
+ fn default_metric_types() -> Vec<MetricType> {
+ vec![MetricType::SUMMARY, MetricType::DEV]
+ }
+
/// Create a wrapper around an [`ExecutionPlan`] which can be
/// pretty printed in a variety of ways
pub fn new(inner: &'a dyn ExecutionPlan) -> Self {
@@ -133,6 +140,7 @@ impl<'a> DisplayableExecutionPlan<'a> {
show_metrics: ShowMetrics::None,
show_statistics: false,
show_schema: false,
+ metric_types: Self::default_metric_types(),
tree_maximum_render_width: 240,
}
}
@@ -146,6 +154,7 @@ impl<'a> DisplayableExecutionPlan<'a> {
show_metrics: ShowMetrics::Aggregated,
show_statistics: false,
show_schema: false,
+ metric_types: Self::default_metric_types(),
tree_maximum_render_width: 240,
}
}
@@ -159,6 +168,7 @@ impl<'a> DisplayableExecutionPlan<'a> {
show_metrics: ShowMetrics::Full,
show_statistics: false,
show_schema: false,
+ metric_types: Self::default_metric_types(),
tree_maximum_render_width: 240,
}
}
@@ -178,6 +188,12 @@ impl<'a> DisplayableExecutionPlan<'a> {
self
}
+ /// Specify which metric types should be rendered alongside the plan
+ pub fn set_metric_types(mut self, metric_types: Vec<MetricType>) -> Self {
+ self.metric_types = metric_types;
+ self
+ }
+
/// Set the maximum render width for the tree format
pub fn set_tree_maximum_render_width(mut self, width: usize) -> Self {
self.tree_maximum_render_width = width;
@@ -206,6 +222,7 @@ impl<'a> DisplayableExecutionPlan<'a> {
show_metrics: ShowMetrics,
show_statistics: bool,
show_schema: bool,
+ metric_types: Vec<MetricType>,
}
impl fmt::Display for Wrapper<'_> {
fn fmt(&self, f: &mut Formatter) -> fmt::Result {
@@ -216,6 +233,7 @@ impl<'a> DisplayableExecutionPlan<'a> {
show_metrics: self.show_metrics,
show_statistics: self.show_statistics,
show_schema: self.show_schema,
+ metric_types: &self.metric_types,
};
accept(self.plan, &mut visitor)
}
@@ -226,6 +244,7 @@ impl<'a> DisplayableExecutionPlan<'a> {
show_metrics: self.show_metrics,
show_statistics: self.show_statistics,
show_schema: self.show_schema,
+ metric_types: self.metric_types.clone(),
}
}
@@ -245,6 +264,7 @@ impl<'a> DisplayableExecutionPlan<'a> {
plan: &'a dyn ExecutionPlan,
show_metrics: ShowMetrics,
show_statistics: bool,
+ metric_types: Vec<MetricType>,
}
impl fmt::Display for Wrapper<'_> {
fn fmt(&self, f: &mut Formatter) -> fmt::Result {
@@ -255,6 +275,7 @@ impl<'a> DisplayableExecutionPlan<'a> {
t,
show_metrics: self.show_metrics,
show_statistics: self.show_statistics,
+ metric_types: &self.metric_types,
graphviz_builder: GraphvizBuilder::default(),
parents: Vec::new(),
};
@@ -272,6 +293,7 @@ impl<'a> DisplayableExecutionPlan<'a> {
plan: self.inner,
show_metrics: self.show_metrics,
show_statistics: self.show_statistics,
+ metric_types: self.metric_types.clone(),
}
}
@@ -306,6 +328,7 @@ impl<'a> DisplayableExecutionPlan<'a> {
show_metrics: ShowMetrics,
show_statistics: bool,
show_schema: bool,
+ metric_types: Vec<MetricType>,
}
impl fmt::Display for Wrapper<'_> {
@@ -317,6 +340,7 @@ impl<'a> DisplayableExecutionPlan<'a> {
show_metrics: self.show_metrics,
show_statistics: self.show_statistics,
show_schema: self.show_schema,
+ metric_types: &self.metric_types,
};
visitor.pre_visit(self.plan)?;
Ok(())
@@ -328,6 +352,7 @@ impl<'a> DisplayableExecutionPlan<'a> {
show_metrics: self.show_metrics,
show_statistics: self.show_statistics,
show_schema: self.show_schema,
+ metric_types: self.metric_types.clone(),
}
}
@@ -382,6 +407,8 @@ struct IndentVisitor<'a, 'b> {
show_statistics: bool,
/// If schema should be displayed
show_schema: bool,
+ /// Which metric types should be rendered
+ metric_types: &'a [MetricType],
}
impl ExecutionPlanVisitor for IndentVisitor<'_, '_> {
@@ -394,6 +421,7 @@ impl ExecutionPlanVisitor for IndentVisitor<'_, '_> {
ShowMetrics::Aggregated => {
if let Some(metrics) = plan.metrics() {
let metrics = metrics
+ .filter_by_metric_types(self.metric_types)
.aggregate_by_name()
.sorted_for_display()
.timestamps_removed();
@@ -405,6 +433,7 @@ impl ExecutionPlanVisitor for IndentVisitor<'_, '_> {
}
ShowMetrics::Full => {
if let Some(metrics) = plan.metrics() {
+ let metrics = metrics.filter_by_metric_types(self.metric_types);
write!(self.f, ", metrics=[{metrics}]")?;
} else {
write!(self.f, ", metrics=[]")?;
@@ -441,6 +470,8 @@ struct GraphvizVisitor<'a, 'b> {
show_metrics: ShowMetrics,
/// If statistics should be displayed
show_statistics: bool,
+ /// Which metric types should be rendered
+ metric_types: &'a [MetricType],
graphviz_builder: GraphvizBuilder,
/// Used to record parent node ids when visiting a plan.
@@ -478,6 +509,7 @@ impl ExecutionPlanVisitor for GraphvizVisitor<'_, '_> {
ShowMetrics::Aggregated => {
if let Some(metrics) = plan.metrics() {
let metrics = metrics
+ .filter_by_metric_types(self.metric_types)
.aggregate_by_name()
.sorted_for_display()
.timestamps_removed();
@@ -489,6 +521,7 @@ impl ExecutionPlanVisitor for GraphvizVisitor<'_, '_> {
}
ShowMetrics::Full => {
if let Some(metrics) = plan.metrics() {
+ let metrics = metrics.filter_by_metric_types(self.metric_types);
format!("metrics=[{metrics}]")
} else {
"metrics=[]".to_string()
diff --git a/datafusion/physical-plan/src/metrics/baseline.rs b/datafusion/physical-plan/src/metrics/baseline.rs
index 15efb8f90a..45cef58b5d 100644
--- a/datafusion/physical-plan/src/metrics/baseline.rs
+++ b/datafusion/physical-plan/src/metrics/baseline.rs
@@ -62,9 +62,15 @@ impl BaselineMetrics {
start_time.record();
Self {
- end_time: MetricBuilder::new(metrics).end_timestamp(partition),
- elapsed_compute: MetricBuilder::new(metrics).elapsed_compute(partition),
- output_rows: MetricBuilder::new(metrics).output_rows(partition),
+ end_time: MetricBuilder::new(metrics)
+ .with_type(super::MetricType::SUMMARY)
+ .end_timestamp(partition),
+ elapsed_compute: MetricBuilder::new(metrics)
+ .with_type(super::MetricType::SUMMARY)
+ .elapsed_compute(partition),
+ output_rows: MetricBuilder::new(metrics)
+ .with_type(super::MetricType::SUMMARY)
+ .output_rows(partition),
}
}
diff --git a/datafusion/physical-plan/src/metrics/builder.rs b/datafusion/physical-plan/src/metrics/builder.rs
index dbda0a310c..74ba5a2a18 100644
--- a/datafusion/physical-plan/src/metrics/builder.rs
+++ b/datafusion/physical-plan/src/metrics/builder.rs
@@ -19,6 +19,8 @@
use std::{borrow::Cow, sync::Arc};
+use crate::metrics::MetricType;
+
use super::{
Count, ExecutionPlanMetricsSet, Gauge, Label, Metric, MetricValue, Time, Timestamp,
};
@@ -52,15 +54,23 @@ pub struct MetricBuilder<'a> {
/// arbitrary name=value pairs identifying this metric
labels: Vec<Label>,
+
+ /// The type controlling the verbosity/category for this builder
+ /// See comments in [`MetricType`] for details
+ metric_type: MetricType,
}
impl<'a> MetricBuilder<'a> {
/// Create a new `MetricBuilder` that will register the result of `build()` with the `metrics`
+ ///
+ /// `self.metric_type` controls when such metric is displayed. See comments in
+ /// [`MetricType`] for details.
pub fn new(metrics: &'a ExecutionPlanMetricsSet) -> Self {
Self {
metrics,
partition: None,
labels: vec![],
+ metric_type: MetricType::DEV,
}
}
@@ -70,6 +80,12 @@ impl<'a> MetricBuilder<'a> {
self
}
+ /// Set the metric type to the metric being constructed
+ pub fn with_type(mut self, metric_type: MetricType) -> Self {
+ self.metric_type = metric_type;
+ self
+ }
+
/// Add a label to the metric being constructed
pub fn with_new_label(
self,
@@ -92,8 +108,11 @@ impl<'a> MetricBuilder<'a> {
labels,
partition,
metrics,
+ metric_type,
} = self;
- let metric = Arc::new(Metric::new_with_labels(value, partition, labels));
+ let metric = Arc::new(
+ Metric::new_with_labels(value, partition, labels).with_type(metric_type),
+ );
metrics.register(metric);
}
diff --git a/datafusion/physical-plan/src/metrics/mod.rs b/datafusion/physical-plan/src/metrics/mod.rs
index 0b9b4bed85..0fd7bfb8c8 100644
--- a/datafusion/physical-plan/src/metrics/mod.rs
+++ b/datafusion/physical-plan/src/metrics/mod.rs
@@ -78,6 +78,29 @@ pub struct Metric {
/// To which partition of an operators output did this metric
/// apply? If `None` then means all partitions.
partition: Option<usize>,
+
+ metric_type: MetricType,
+}
+
+/// Categorizes metrics so the display layer can choose the desired verbosity.
+///
+/// # How is it used:
+/// The `datafusion.explain.analyze_level` configuration controls which category is shown.
+/// - When set to `dev`, all metrics with type `MetricType::Summary` or `MetricType::DEV`
+/// will be shown.
+/// - When set to `summary`, only metrics with type `MetricType::Summary` are shown.
+///
+/// # Difference from `EXPLAIN ANALYZE VERBOSE`:
+/// The `VERBOSE` keyword controls whether per-partition metrics are shown (when specified),
+/// or aggregated metrics are displayed (when omitted).
+/// In contrast, the `analyze_level` configuration determines which categories or
+/// levels of metrics are displayed.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+pub enum MetricType {
+ /// Common metrics for high-level insights (answering which operator is slow)
+ SUMMARY,
+ /// For deep operator-level introspection for developers
+ DEV,
}
impl Display for Metric {
@@ -122,6 +145,7 @@ impl Metric {
value,
labels: vec![],
partition,
+ metric_type: MetricType::DEV,
}
}
@@ -136,9 +160,16 @@ impl Metric {
value,
labels,
partition,
+ metric_type: MetricType::DEV,
}
}
+ /// Set the type for this metric. Defaults to [`MetricType::DEV`]
+ pub fn with_type(mut self, metric_type: MetricType) -> Self {
+ self.metric_type = metric_type;
+ self
+ }
+
/// Add a new label to this metric
pub fn with_label(mut self, label: Label) -> Self {
self.labels.push(label);
@@ -164,6 +195,11 @@ impl Metric {
pub fn partition(&self) -> Option<usize> {
self.partition
}
+
+ /// Return the metric type (verbosity level) associated with this metric
+ pub fn metric_type(&self) -> MetricType {
+ self.metric_type
+ }
}
/// A snapshot of the metrics for a particular ([`ExecutionPlan`]).
@@ -286,7 +322,8 @@ impl MetricsSet {
.or_insert_with(|| {
// accumulate with no partition
let partition = None;
- let mut accum = Metric::new(metric.value().new_empty(), partition);
+ let mut accum = Metric::new(metric.value().new_empty(), partition)
+ .with_type(metric.metric_type());
accum.value_mut().aggregate(metric.value());
accum
});
@@ -324,6 +361,21 @@ impl MetricsSet {
Self { metrics }
}
+
+ /// Returns a new derived `MetricsSet` containing only metrics whose
+ /// [`MetricType`] appears in `allowed`.
+ pub fn filter_by_metric_types(self, allowed: &[MetricType]) -> Self {
+ if allowed.is_empty() {
+ return Self { metrics: vec![] };
+ }
+
+ let metrics = self
+ .metrics
+ .into_iter()
+ .filter(|metric| allowed.contains(&metric.metric_type()))
+ .collect::<Vec<_>>();
+ Self { metrics }
+ }
}
impl Display for MetricsSet {
diff --git a/datafusion/proto/src/physical_plan/mod.rs b/datafusion/proto/src/physical_plan/mod.rs
index d76bcc89b3..818109be68 100644
--- a/datafusion/proto/src/physical_plan/mod.rs
+++ b/datafusion/proto/src/physical_plan/mod.rs
@@ -82,6 +82,7 @@ use datafusion::physical_plan::joins::{
use datafusion::physical_plan::joins::{HashJoinExec, PartitionMode};
use datafusion::physical_plan::limit::{GlobalLimitExec, LocalLimitExec};
use datafusion::physical_plan::memory::LazyMemoryExec;
+use datafusion::physical_plan::metrics::MetricType;
use datafusion::physical_plan::placeholder_row::PlaceholderRowExec;
use datafusion::physical_plan::projection::{ProjectionExec, ProjectionExpr};
use datafusion::physical_plan::repartition::RepartitionExec;
@@ -1611,6 +1612,7 @@ impl protobuf::PhysicalPlanNode {
Ok(Arc::new(AnalyzeExec::new(
analyze.verbose,
analyze.show_statistics,
+ vec![MetricType::SUMMARY, MetricType::DEV],
input,
Arc::new(convert_required!(analyze.schema)?),
)))
diff --git a/datafusion/proto/tests/cases/roundtrip_physical_plan.rs b/datafusion/proto/tests/cases/roundtrip_physical_plan.rs
index b93d0d3c4e..e6cfcb9580 100644
--- a/datafusion/proto/tests/cases/roundtrip_physical_plan.rs
+++ b/datafusion/proto/tests/cases/roundtrip_physical_plan.rs
@@ -32,6 +32,7 @@ use arrow::csv::WriterBuilder;
use arrow::datatypes::{Fields, TimeUnit};
use datafusion::physical_expr::aggregate::AggregateExprBuilder;
use datafusion::physical_plan::coalesce_batches::CoalesceBatchesExec;
+use datafusion::physical_plan::metrics::MetricType;
use datafusion_expr::dml::InsertOp;
use datafusion_functions_aggregate::approx_percentile_cont::approx_percentile_cont_udaf;
use datafusion_functions_aggregate::array_agg::array_agg_udaf;
@@ -1436,6 +1437,7 @@ fn roundtrip_analyze() -> Result<()> {
roundtrip_test(Arc::new(AnalyzeExec::new(
false,
false,
+ vec![MetricType::SUMMARY, MetricType::DEV],
input,
Arc::new(schema),
)))
diff --git a/datafusion/sqllogictest/test_files/information_schema.slt b/datafusion/sqllogictest/test_files/information_schema.slt
index a69a8d5c0d..412e36b812 100644
--- a/datafusion/sqllogictest/test_files/information_schema.slt
+++ b/datafusion/sqllogictest/test_files/information_schema.slt
@@ -269,6 +269,7 @@ datafusion.execution.split_file_groups_by_statistics false
datafusion.execution.target_partitions 7
datafusion.execution.time_zone +00:00
datafusion.execution.use_row_number_estimates_to_optimize_partitioning false
+datafusion.explain.analyze_level dev
datafusion.explain.format indent
datafusion.explain.logical_plan_only false
datafusion.explain.physical_plan_only false
@@ -387,6 +388,7 @@ datafusion.execution.split_file_groups_by_statistics false Attempt to eliminate
datafusion.execution.target_partitions 7 Number of partitions for query execution. Increasing partitions can increase concurrency. Defaults to the number of CPU cores on the system
datafusion.execution.time_zone +00:00 The default time zone Some functions, e.g. `EXTRACT(HOUR from SOME_TIME)`, shift the underlying datetime according to this time zone, and then extract the hour
datafusion.execution.use_row_number_estimates_to_optimize_partitioning false Should DataFusion use row number estimates at the input to decide whether increasing parallelism is beneficial or not. By default, only exact row numbers (not estimates) are used for this decision. Setting this flag to `true` will likely produce better plans. if the source of statistics is accurate. We plan to make this the default in the future.
+datafusion.explain.analyze_level dev Verbosity level for "EXPLAIN ANALYZE". Default is "dev" "summary" shows common metrics for high-level insights. "dev" provides deep operator-level introspection for developers.
datafusion.explain.format indent Display format of explain. Default is "indent". When set to "tree", it will print the plan in a tree-rendered format.
datafusion.explain.logical_plan_only false When set to true, the explain statement will only print logical plans
datafusion.explain.physical_plan_only false When set to true, the explain statement will only print physical plans
diff --git a/datafusion/sqllogictest/test_files/spark/aggregate/avg.slt b/datafusion/sqllogictest/test_files/spark/aggregate/avg.slt
index a5bed6ea32..6ae647989a 100644
--- a/datafusion/sqllogictest/test_files/spark/aggregate/avg.slt
+++ b/datafusion/sqllogictest/test_files/spark/aggregate/avg.slt
@@ -53,4 +53,4 @@ FROM (VALUES (0::INT), (0::INT)) AS t(a)
GROUP BY a
ORDER BY a;
----
-0 0
\ No newline at end of file
+0 0
diff --git a/datafusion/sqllogictest/test_files/spark/string/concat.slt b/datafusion/sqllogictest/test_files/spark/string/concat.slt
index 0b796a54a6..258cb829d7 100644
--- a/datafusion/sqllogictest/test_files/spark/string/concat.slt
+++ b/datafusion/sqllogictest/test_files/spark/string/concat.slt
@@ -45,4 +45,4 @@ query T
SELECT concat(a, b, c) from (select 'a' a, 'b' b, 'c' c union all select null a, 'b', 'c') order by 1 nulls last;
----
abc
-NULL
\ No newline at end of file
+NULL
diff --git a/datafusion/sqllogictest/test_files/spark/string/format_string.slt b/datafusion/sqllogictest/test_files/spark/string/format_string.slt
index 07c8cd10d1..048863ebfb 100644
--- a/datafusion/sqllogictest/test_files/spark/string/format_string.slt
+++ b/datafusion/sqllogictest/test_files/spark/string/format_string.slt
@@ -70,7 +70,7 @@ SELECT format_string('Value: %d', 42);
Value: 42
## Hexadecimal integer formatting (lowercase)
-query T
+query T
SELECT format_string('Hex: %x', 255);
----
Hex: ff
@@ -118,7 +118,7 @@ SELECT format_string('Negative: %d', -42);
Negative: -42
# ================================
-# Float formatting tests
+# Float formatting tests
# ================================
## Basic float formatting
@@ -254,7 +254,7 @@ SELECT format_string('Bool: %b', true);
----
Bool: true
-## Boolean uppercase
+## Boolean uppercase
query T
SELECT format_string('Bool: %B', false);
----
@@ -370,7 +370,7 @@ SELECT format_string('Minute: %tM', TIMESTAMP '2023-12-25 14:30:45');
----
Minute: 30
-## Second formatting
+## Second formatting
query T
SELECT format_string('Second: %tS', TIMESTAMP '2023-12-25 14:30:45');
----
diff --git a/docs/source/user-guide/configs.md b/docs/source/user-guide/configs.md
index ab3b11a8d8..a302cecfa6 100644
--- a/docs/source/user-guide/configs.md
+++ b/docs/source/user-guide/configs.md
@@ -160,6 +160,7 @@ The following configuration settings are available:
| datafusion.explain.show_schema | false | When set to true, the explain statement will print schema information [...]
| datafusion.explain.format | indent | Display format of explain. Default is "indent". When set to "tree", it will print the plan in a tree-rendered format. [...]
| datafusion.explain.tree_maximum_render_width | 240 | (format=tree only) Maximum total width of the rendered tree. When set to 0, the tree will have no width limit. [...]
+| datafusion.explain.analyze_level | dev | Verbosity level for "EXPLAIN ANALYZE". Default is "dev" "summary" shows common metrics for high-level insights. "dev" provides deep operator-level introspection for developers. [...]
| datafusion.sql_parser.parse_float_as_decimal | false | When set to true, SQL parser will parse float as decimal type [...]
| datafusion.sql_parser.enable_ident_normalization | true | When set to true, SQL parser will normalize ident (convert ident to lowercase when not quoted) [...]
| datafusion.sql_parser.enable_options_value_normalization | false | When set to true, SQL parser will normalize options value (convert value to lowercase). Note that this option is ignored and will be removed in the future. All case-insensitive values are normalized automatically. [...]
diff --git a/docs/source/user-guide/sql/explain.md b/docs/source/user-guide/sql/explain.md
index c5e2e215a6..1caadcc291 100644
--- a/docs/source/user-guide/sql/explain.md
+++ b/docs/source/user-guide/sql/explain.md
@@ -37,8 +37,6 @@ The optional `[FORMAT format]` clause controls how the plan is displayed as
explained below. If this clause is not specified, the plan is displayed using
the format from the [configuration value] `datafusion.explain.format`.
-[configuration value]: ../configs.md
-
### `tree` format (default)
The `tree` format is modeled after [DuckDB plans] and is designed to be easier
@@ -239,8 +237,7 @@ Elapsed 0.010 seconds.
## `EXPLAIN ANALYZE`
-Shows the execution plan and metrics of a statement. If you need more
-information output, use `EXPLAIN ANALYZE VERBOSE`. Note that `EXPLAIN ANALYZE`
+Shows the execution plan and metrics of a statement. Note that `EXPLAIN ANALYZE`
only supports the `indent` format.
```sql
@@ -259,3 +256,9 @@ EXPLAIN ANALYZE SELECT SUM(x) FROM table GROUP BY b;
| | DataSourceExec: file_groups={1 group: [[/tmp/table.csv]]}, has_header=false, metrics=[] |
+-------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+
```
+
+By default `EXPLAIN ANALYZE` shows the aggregated metrics from all partitions for each operator. If you need to display per-partition metrics, use `EXPLAIN ANALYZE VERBOSE`.
+
+You can also set `datafusion.explain.analyze_level` from the [configuration value] to control the detail level for the metrics displayed.
+
+[configuration value]: ../configs.md
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org