You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "findepi (via GitHub)" <gi...@apache.org> on 2025/09/18 15:50:11 UTC

[PR] Display qualifiers in EXPLAIN [datafusion]

findepi opened a new pull request, #17645:
URL: https://github.com/apache/datafusion/pull/17645

   Sometimes we display qualifiers in plan's Display, e.g. for `Column`, but sometimes not. This adds qualifiers to output for `SubqueryAlias` and in `LogicalPlan::display_indent_schema`. Qualifiers are sometimes necessary to understand the plan semantics, especially when dealing with duplicate names, e.g. in joins.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Display qualifiers in EXPLAIN [datafusion]

Posted by "findepi (via GitHub)" <gi...@apache.org>.
findepi commented on code in PR #17645:
URL: https://github.com/apache/datafusion/pull/17645#discussion_r2383479380


##########
datafusion/core/tests/sql/explain_analyze.rs:
##########
@@ -182,9 +182,9 @@ async fn csv_explain_plans() {
         actual,
         @r"
     Explain [plan_type:Utf8, plan:Utf8]
-      Projection: aggregate_test_100.c1 [c1:Utf8View]
-        Filter: aggregate_test_100.c2 > Int64(10) [c1:Utf8View, c2:Int8, c3:Int16, c4:Int16, c5:Int32, c6:Int64, c7:Int16, c8:Int32, c9:UInt32, c10:UInt64, c11:Float32, c12:Float64, c13:Utf8View]
-          TableScan: aggregate_test_100 [c1:Utf8View, c2:Int8, c3:Int16, c4:Int16, c5:Int32, c6:Int64, c7:Int16, c8:Int32, c9:UInt32, c10:UInt64, c11:Float32, c12:Float64, c13:Utf8View]
+      Projection: aggregate_test_100.c1 [aggregate_test_100.c1:Utf8View]
+        Filter: aggregate_test_100.c2 > Int64(10) [aggregate_test_100.c1:Utf8View, aggregate_test_100.c2:Int8, aggregate_test_100.c3:Int16, aggregate_test_100.c4:Int16, aggregate_test_100.c5:Int32, aggregate_test_100.c6:Int64, aggregate_test_100.c7:Int16, aggregate_test_100.c8:Int32, aggregate_test_100.c9:UInt32, aggregate_test_100.c10:UInt64, aggregate_test_100.c11:Float32, aggregate_test_100.c12:Float64, aggregate_test_100.c13:Utf8View]

Review Comment:
   is this because all fields are qualified and all have the same qualifier?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Display qualifiers in EXPLAIN [datafusion]

Posted by "findepi (via GitHub)" <gi...@apache.org>.
findepi closed pull request #17645: Display qualifiers in EXPLAIN
URL: https://github.com/apache/datafusion/pull/17645


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Display qualifiers in EXPLAIN [datafusion]

Posted by "findepi (via GitHub)" <gi...@apache.org>.
findepi commented on code in PR #17645:
URL: https://github.com/apache/datafusion/pull/17645#discussion_r2362710967


##########
datafusion/core/tests/dataframe/dataframe_functions.rs:
##########
@@ -1310,8 +1310,8 @@ async fn test_count_wildcard() -> Result<()> {
         @r"
     Sort: count(*) ASC NULLS LAST [count(*):Int64]
       Projection: count(*) [count(*):Int64]
-        Aggregate: groupBy=[[test.b]], aggr=[[count(Int64(1)) AS count(*)]] [b:UInt32, count(*):Int64]
-          TableScan: test [a:UInt32, b:UInt32, c:UInt32]
+        Aggregate: groupBy=[[test.b]], aggr=[[count(Int64(1)) AS count(*)]] [test.b:UInt32, count(*):Int64]
+          TableScan: test [test.a:UInt32, test.b:UInt32, test.c:UInt32]

Review Comment:
   For TableScan, however. the schema printing code is same for every plan node and for many it's not much less clear. Without this change, the plan printout is incomplete and insufficient to understand the plan.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Display qualifiers in EXPLAIN [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #17645:
URL: https://github.com/apache/datafusion/pull/17645#discussion_r2380049014


##########
datafusion/core/tests/dataframe/mod.rs:
##########
@@ -1863,7 +1863,7 @@ async fn with_column_renamed_join() -> Result<()> {
     assert_snapshot!(
         df_renamed.logical_plan(),
         @r"
-    Projection: t1.c1 AS AAA, t1.c2, t1.c3, t2.c1, t2.c2, t2.c3
+    Projection: t1.c1 AS t1.AAA, t1.c2, t1.c3, t2.c1, t2.c2, t2.c3

Review Comment:
   This doesn't seem right to me -- the alias shouldn't have a qualifier on it, should it? `AAA` doesn't come from the `t1` relation, it is created in the outer query 



##########
datafusion/core/tests/sql/explain_analyze.rs:
##########
@@ -182,9 +182,9 @@ async fn csv_explain_plans() {
         actual,
         @r"
     Explain [plan_type:Utf8, plan:Utf8]
-      Projection: aggregate_test_100.c1 [c1:Utf8View]
-        Filter: aggregate_test_100.c2 > Int64(10) [c1:Utf8View, c2:Int8, c3:Int16, c4:Int16, c5:Int32, c6:Int64, c7:Int16, c8:Int32, c9:UInt32, c10:UInt64, c11:Float32, c12:Float64, c13:Utf8View]
-          TableScan: aggregate_test_100 [c1:Utf8View, c2:Int8, c3:Int16, c4:Int16, c5:Int32, c6:Int64, c7:Int16, c8:Int32, c9:UInt32, c10:UInt64, c11:Float32, c12:Float64, c13:Utf8View]
+      Projection: aggregate_test_100.c1 [aggregate_test_100.c1:Utf8View]
+        Filter: aggregate_test_100.c2 > Int64(10) [aggregate_test_100.c1:Utf8View, aggregate_test_100.c2:Int8, aggregate_test_100.c3:Int16, aggregate_test_100.c4:Int16, aggregate_test_100.c5:Int32, aggregate_test_100.c6:Int64, aggregate_test_100.c7:Int16, aggregate_test_100.c8:Int32, aggregate_test_100.c9:UInt32, aggregate_test_100.c10:UInt64, aggregate_test_100.c11:Float32, aggregate_test_100.c12:Float64, aggregate_test_100.c13:Utf8View]

Review Comment:
   this is a good example of a plan which is much less readable after this change in my mind 



##########
datafusion/core/tests/dataframe/mod.rs:
##########
@@ -3606,12 +3606,12 @@ async fn join_with_alias_filter() -> Result<()> {
     let actual = formatted.trim();
     assert_snapshot!(
         actual,
-        @r###"
-    Projection: t1.a, t2.a, t1.b, t1.c, t2.b, t2.c [a:UInt32, a:UInt32, b:Utf8, c:Int32, b:Utf8, c:Int32]
-      Inner Join: t1.a + UInt32(3) = t2.a + UInt32(1) [a:UInt32, b:Utf8, c:Int32, a:UInt32, b:Utf8, c:Int32]
-        TableScan: t1 projection=[a, b, c] [a:UInt32, b:Utf8, c:Int32]
-        TableScan: t2 projection=[a, b, c] [a:UInt32, b:Utf8, c:Int32]
-    "###
+        @r"
+    Projection: t1.a, t2.a, t1.b, t1.c, t2.b, t2.c [t1.a:UInt32, t2.a:UInt32, t1.b:Utf8, t1.c:Int32, t2.b:Utf8, t2.c:Int32]

Review Comment:
   I think it is an improvement for the `Projection` and `Inner Join` here to have the qualifiers on them -- that makes them less ambiguous when there are potentially multiple relations



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Display qualifiers in EXPLAIN [datafusion]

Posted by "xudong963 (via GitHub)" <gi...@apache.org>.
xudong963 commented on code in PR #17645:
URL: https://github.com/apache/datafusion/pull/17645#discussion_r2362192246


##########
datafusion/core/tests/dataframe/dataframe_functions.rs:
##########
@@ -1310,8 +1310,8 @@ async fn test_count_wildcard() -> Result<()> {
         @r"
     Sort: count(*) ASC NULLS LAST [count(*):Int64]
       Projection: count(*) [count(*):Int64]
-        Aggregate: groupBy=[[test.b]], aggr=[[count(Int64(1)) AS count(*)]] [b:UInt32, count(*):Int64]
-          TableScan: test [a:UInt32, b:UInt32, c:UInt32]
+        Aggregate: groupBy=[[test.b]], aggr=[[count(Int64(1)) AS count(*)]] [test.b:UInt32, count(*):Int64]
+          TableScan: test [test.a:UInt32, test.b:UInt32, test.c:UInt32]

Review Comment:
   Does the `test` already tell us the qualifier?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Display qualifiers in EXPLAIN [datafusion]

Posted by "findepi (via GitHub)" <gi...@apache.org>.
findepi commented on code in PR #17645:
URL: https://github.com/apache/datafusion/pull/17645#discussion_r2383481367


##########
datafusion/core/tests/dataframe/dataframe_functions.rs:
##########
@@ -1310,8 +1310,8 @@ async fn test_count_wildcard() -> Result<()> {
         @r"
     Sort: count(*) ASC NULLS LAST [count(*):Int64]
       Projection: count(*) [count(*):Int64]
-        Aggregate: groupBy=[[test.b]], aggr=[[count(Int64(1)) AS count(*)]] [b:UInt32, count(*):Int64]
-          TableScan: test [a:UInt32, b:UInt32, c:UInt32]
+        Aggregate: groupBy=[[test.b]], aggr=[[count(Int64(1)) AS count(*)]] [test.b:UInt32, count(*):Int64]
+          TableScan: test [test.a:UInt32, test.b:UInt32, test.c:UInt32]

Review Comment:
   > More generally, when there is only one relation in the query, as is the case in many queries, adding a qualifier to all expressions I think makes the plans harder to read, not better
   
   Agreed.
   But also, single-table queries are not the ones we should optimize EXPLAIN output for.
   These represent a subset of all queries which naturally is simpler than all queries, without source table count limit.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Display qualifiers in EXPLAIN [datafusion]

Posted by "findepi (via GitHub)" <gi...@apache.org>.
findepi commented on code in PR #17645:
URL: https://github.com/apache/datafusion/pull/17645#discussion_r2369029404


##########
datafusion/core/tests/dataframe/dataframe_functions.rs:
##########
@@ -1310,8 +1310,8 @@ async fn test_count_wildcard() -> Result<()> {
         @r"
     Sort: count(*) ASC NULLS LAST [count(*):Int64]
       Projection: count(*) [count(*):Int64]
-        Aggregate: groupBy=[[test.b]], aggr=[[count(Int64(1)) AS count(*)]] [b:UInt32, count(*):Int64]
-          TableScan: test [a:UInt32, b:UInt32, c:UInt32]
+        Aggregate: groupBy=[[test.b]], aggr=[[count(Int64(1)) AS count(*)]] [test.b:UInt32, count(*):Int64]
+          TableScan: test [test.a:UInt32, test.b:UInt32, test.c:UInt32]

Review Comment:
   Could that be confusing? If some qualifiers are printed but some not, the projections without qualifiers will look as if they did not have any, which is a different state from the one when they all have the same qualifier.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Display qualifiers in EXPLAIN [datafusion]

Posted by "findepi (via GitHub)" <gi...@apache.org>.
findepi commented on code in PR #17645:
URL: https://github.com/apache/datafusion/pull/17645#discussion_r2383478286


##########
datafusion/core/tests/dataframe/mod.rs:
##########
@@ -1863,7 +1863,7 @@ async fn with_column_renamed_join() -> Result<()> {
     assert_snapshot!(
         df_renamed.logical_plan(),
         @r"
-    Projection: t1.c1 AS AAA, t1.c2, t1.c3, t2.c1, t2.c2, t2.c3
+    Projection: t1.c1 AS t1.AAA, t1.c2, t1.c3, t2.c1, t2.c2, t2.c3

Review Comment:
   I honestly have no idea where `t1.` comes from, and what should be here.
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Display qualifiers in EXPLAIN [datafusion]

Posted by "findepi (via GitHub)" <gi...@apache.org>.
findepi commented on PR #17645:
URL: https://github.com/apache/datafusion/pull/17645#issuecomment-3340499323

   I see how it's controversial. Maybe it could go behind a  session property.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Display qualifiers in EXPLAIN [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #17645:
URL: https://github.com/apache/datafusion/pull/17645#discussion_r2380054357


##########
datafusion/core/tests/dataframe/dataframe_functions.rs:
##########
@@ -1310,8 +1310,8 @@ async fn test_count_wildcard() -> Result<()> {
         @r"
     Sort: count(*) ASC NULLS LAST [count(*):Int64]
       Projection: count(*) [count(*):Int64]
-        Aggregate: groupBy=[[test.b]], aggr=[[count(Int64(1)) AS count(*)]] [b:UInt32, count(*):Int64]
-          TableScan: test [a:UInt32, b:UInt32, c:UInt32]
+        Aggregate: groupBy=[[test.b]], aggr=[[count(Int64(1)) AS count(*)]] [test.b:UInt32, count(*):Int64]
+          TableScan: test [test.a:UInt32, test.b:UInt32, test.c:UInt32]

Review Comment:
   I was more thinking how redundant this line is now
   
   It goes from
   ```diff
   -          TableScan: test [a:UInt32, b:UInt32, c:UInt32]
   +          TableScan: test [test.a:UInt32, test.b:UInt32, test.c:UInt32]
   ```
   
   That is the qualifier `test` is now repeated **4** times. It will be even worse when there are
   1. long qualifiers "my_really_obxiously_long_table_name"
   2. Multiple columns selected as each column gets the same name 
   
   For a TableScan, there can be, by definition, only a single relation, so appending the relation name to all expressions just makes the plans harder to read
   
   
   More generally, when there is only one relation in the query, as is the case in many queries, adding a qualifier to all expressions I think makes the plans harder to read, not better
   
   ```
   
   
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Display qualifiers in EXPLAIN [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #17645:
URL: https://github.com/apache/datafusion/pull/17645#discussion_r2364107262


##########
datafusion/core/tests/dataframe/dataframe_functions.rs:
##########
@@ -1310,8 +1310,8 @@ async fn test_count_wildcard() -> Result<()> {
         @r"
     Sort: count(*) ASC NULLS LAST [count(*):Int64]
       Projection: count(*) [count(*):Int64]
-        Aggregate: groupBy=[[test.b]], aggr=[[count(Int64(1)) AS count(*)]] [b:UInt32, count(*):Int64]
-          TableScan: test [a:UInt32, b:UInt32, c:UInt32]
+        Aggregate: groupBy=[[test.b]], aggr=[[count(Int64(1)) AS count(*)]] [test.b:UInt32, count(*):Int64]
+          TableScan: test [test.a:UInt32, test.b:UInt32, test.c:UInt32]

Review Comment:
   Maybe we can special case the schema printing code to have a version to skip the qualifiers in cases where it is always the same 🤔 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org