alamb opened a new issue, #15343: URL: https://github.com/apache/datafusion/issues/15343 ### Is your feature request related to a problem or challenge? _No response_ ### Describe the solution you'd like _No response_ ### Describe alternatives you've considered _No response_ ### Additional context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org For queries about this service, please contact Infrastructure at: users@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org For additional commands, e-mail: github-help@datafusion.apache.org
Weijun-H commented on issue #15343: URL: https://github.com/apache/datafusion/issues/15343#issuecomment-2747560863 I agree with this change. The `tree` format works well for the explanation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org For queries about this service, please contact Infrastructure at: users@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org For additional commands, e-mail: github-help@datafusion.apache.org
ozankabak commented on issue #15343: URL: https://github.com/apache/datafusion/issues/15343#issuecomment-2745139974 > I wonder if we could just change the default format for `datafusion-cli` (it is a config setting) π€ Downstream projects could also then "opt-in" if they wanted nicer default explain plans Makes sense to change the default format for `datafusion-cli` π -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org For queries about this service, please contact Infrastructure at: users@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org For additional commands, e-mail: github-help@datafusion.apache.org
adriangb commented on issue #15343: URL: https://github.com/apache/datafusion/issues/15343#issuecomment-2742171861 Plus one for making it the default :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org For queries about this service, please contact Infrastructure at: users@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org For additional commands, e-mail: github-help@datafusion.apache.org
Omega359 commented on issue #15343: URL: https://github.com/apache/datafusion/issues/15343#issuecomment-2743786707 I agree with others on the tests, let them continue to use the condensed format. Otherwise I agree with making the tree format the default. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org For queries about this service, please contact Infrastructure at: users@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org For additional commands, e-mail: github-help@datafusion.apache.org
berkaysynnada commented on issue #15343: URL: https://github.com/apache/datafusion/issues/15343#issuecomment-2743413455 > I think this is _very_ important. We found many bugs using the current, detailed plan print-outs. Let's make the tree mode default for the CLI experience (and tests verifying the CLI behavior), but all the other tests should keep using the current format. I agree with this, test coverage should not be weakened. Unless we ensure tree format displays at least as much detail as the ident format, test default should be ident format. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org For queries about this service, please contact Infrastructure at: users@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org For additional commands, e-mail: github-help@datafusion.apache.org
zhuqi-lucas commented on issue #15343: URL: https://github.com/apache/datafusion/issues/15343#issuecomment-2742742054 +1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org For queries about this service, please contact Infrastructure at: users@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org For additional commands, e-mail: github-help@datafusion.apache.org
alamb commented on issue #15343: URL: https://github.com/apache/datafusion/issues/15343#issuecomment-2753832474 Here is a proposed PR: - https://github.com/apache/datafusion/pull/15427 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org For queries about this service, please contact Infrastructure at: users@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org For additional commands, e-mail: github-help@datafusion.apache.org
ozankabak commented on issue #15343: URL: https://github.com/apache/datafusion/issues/15343#issuecomment-2743148343 > Most tests that use the indent mode should continue to use the indent mode as it contains more information. In other words the tests should not be updated to use the tree mode I think this is *very* important. We found many bugs using the current, detailed plan print-outs. Let's make the tree mode default for the CLI experience (and tests verifying the CLI behavior), but all the other tests should keep using the current format. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org For queries about this service, please contact Infrastructure at: users@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org For additional commands, e-mail: github-help@datafusion.apache.org
alamb commented on issue #15343: URL: https://github.com/apache/datafusion/issues/15343#issuecomment-2745139051 I wonder if we could just change the default format for `datafusion-cli` (it is a config setting) π€ Downstream projects could also then "opt-in" if they wanted nicer default explain plans -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org For queries about this service, please contact Infrastructure at: users@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org For additional commands, e-mail: github-help@datafusion.apache.org
alamb closed issue #15343: [DISCUSS] Switch to `tree` explain by default URL: https://github.com/apache/datafusion/issues/15343 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org For queries about this service, please contact Infrastructure at: users@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org For additional commands, e-mail: github-help@datafusion.apache.org
alamb commented on issue #15343:
URL: https://github.com/apache/datafusion/issues/15343#issuecomment-2741964039
For example, here is the plan from a recent query from https://github.com/apache/datafusion/issues/15177 (I actually had to trim it to fit in the 65k limit):
```sql
> explain SELECT * from 'hits_partitioned' WHERE "WatchID" IN (SELECT "WatchID" FROM 'hits_partitioned' WHERE "URL" LIKE '%google%' ORDER BY "EventTime" LIMIT 10);
+---------------+---------
| plan_type | plan
|
+---------------+----------
| logical_plan | LeftSemi Join: hits_partitioned.WatchID = __correlated_sq_1.WatchID
|
| | TableScan: ...
|
| | SubqueryAlias: __correlated_sq_1
|
| | Projection: hits_partitioned.WatchID
|
| | Sort: hits_partitioned.EventTime ASC NULLS LAST, fetch=10
|
| | Projection: hits_partitioned.WatchID, hits_partitioned.EventTime
|
| | Filter: CAST(hits_partitioned.URL AS Utf8View) LIKE Utf8View("%google%")
|
| | TableScan: hits_partitioned projection=[WatchID, EventTime, URL], partial_filters=[CAST(hits_partitioned.URL AS Utf8View) LIKE Utf8View("%google%")]
|
| physical_plan | CoalesceBatchesExec: target_batch_size=8192
|
| | HashJoinExec: mode=Partitioned, join_type=RightSemi, on=[(WatchID@0, WatchID@0)]
|
| | CoalesceBatchesExec: target_batch_size=8192
|
| | RepartitionExec: partitioning=Hash([WatchID@0], 16), input_partitions=16
|
| | RepartitionExec: partitioning=RoundRobinBatch(16), input_partitions=1
|
| | ProjectionExec: expr=[WatchID@0 as WatchID]
|
| | SortPreservingMergeExec: [EventTime@1 ASC NULLS LAST], fetch=10
|
| | SortExec: TopK(fetch=10), expr=[EventTime@1 ASC NULLS LAST], preserve_partitioning=[true]
|
| | CoalesceBatchesExec: target_batch_size=8192
|
| | FilterExec: CAST(URL@2 AS Utf8View) LIKE %google%, projection=[WatchID@0, EventTime@1]
|
| | DataSourceExec: file_groups={16 groups: ...
|
| | CoalesceBatchesExec: target_batch_size=8192
|
| | RepartitionExec: partitioning=Hash([WatchID@0], 16), input_partitions=16
|
| | DataSourceExec: file_groups={16 groups: ...file_type=parquet |
| |
|
+---------------+--
2 row(s) fetched.
Elapsed 0.072 seconds.
```
```sql
EXPLAIN FORMAT tree ...
```
I think the `tree` explain format for the same query is much easier to understand:
```sql
> explain format tree SELECT * from 'hits_partitioned' WHERE "WatchID" IN (SELECT "WatchID" FROM 'hits_partitioned' WHERE "URL" LIKE '%google%' ORDER BY "EventTime" LIMIT 10);
+---------------+------------------------------------------------------------+
| plan_type | plan |
+---------------+------------------------------------------------------------+
| physical_plan | βββββββββββββββββββββββββββββ |
| | β CoalesceBatchesExec β |
| | β -------------------- β |
| | β target_batch_size: β |
| | β 8192 β |
| | βββββββββββββββ¬ββββββββββββββ |
| | βββββββββββββββ΄ββββββββββββββ |
| | β HashJoinExec β |
| | β -------------------- β |
| | β join_type: RightSemi β |
| | β ββββββββββββββββ |
| | β on: β β |
| | β (WatchID = WatchID) β β |
| | βββββββββββββββ¬ββββββββββββββ β |
| | βββββββββββββββ΄βββββββββββββββββββββββββββββ΄ββββββββββββββ |
| | β CoalesceBatchesExec ββ CoalesceBatchesExec β |
| | β -------------------- ββ -------------------- β |
| | β target_batch_size: ββ target_batch_size: β |
| | β 8192 ββ 8192 β |
| | βββββββββββββββ¬βββββββββββββββββββββββββββββ¬ββββββββββββββ |
| | βββββββββββββββ΄βββββββββββββββββββββββββββββ΄ββββββββββββββ |
| | β RepartitionExec ββ RepartitionExec β |
| | β -------------------- ββ -------------------- β |
| | β output_partition_count: ββ output_partition_count: β |
| | β 16 ββ 16 β |
| | β ββ β |
| | β partitioning_scheme: ββ partitioning_scheme: β |
| | β Hash([WatchID@0], 16) ββ Hash([WatchID@0], 16) β |
| | βββββββββββββββ¬βββββββββββββββββββββββββββββ¬ββββββββββββββ |
| | βββββββββββββββ΄βββββββββββββββββββββββββββββ΄ββββββββββββββ |
| | β RepartitionExec ββ DataSourceExec β |
| | β -------------------- ββ -------------------- β |
| | β output_partition_count: ββ files: 115 β |
| | β 1 ββ format: parquet β |
| | β ββ β |
| | β partitioning_scheme: ββ β |
| | β RoundRobinBatch(16) ββ β |
| | βββββββββββββββ¬βββββββββββββββββββββββββββββββββββββββββββ |
| | βββββββββββββββ΄ββββββββββββββ |
| | β ProjectionExec β |
| | β -------------------- β |
| | β WatchID: WatchID β |
| | βββββββββββββββ¬ββββββββββββββ |
| | βββββββββββββββ΄ββββββββββββββ |
| | β SortPreservingMergeExec β |
| | β -------------------- β |
| | β EventTime ASC NULLS β |
| | β LASTlimit: β |
| | β 10 β |
| | βββββββββββββββ¬ββββββββββββββ |
| | βββββββββββββββ΄ββββββββββββββ |
| | β SortExec β |
| | β -------------------- β |
| | β EventTime@1 ASC NULLS LASTβ |
| | β β |
| | β limit: 10 β |
| | βββββββββββββββ¬ββββββββββββββ |
| | βββββββββββββββ΄ββββββββββββββ |
| | β CoalesceBatchesExec β |
| | β -------------------- β |
| | β target_batch_size: β |
| | β 8192 β |
| | βββββββββββββββ¬ββββββββββββββ |
| | βββββββββββββββ΄ββββββββββββββ |
| | β FilterExec β |
| | β -------------------- β |
| | β predicate: β |
| | β CAST(URL AS Utf8View) LIKEβ |
| | β %google% β |
| | βββββββββββββββ¬ββββββββββββββ |
| | βββββββββββββββ΄ββββββββββββββ |
| | β DataSourceExec β |
| | β -------------------- β |
| | β files: 115 β |
| | β format: parquet β |
| | β β |
| | β predicate: β |
| | β CAST(URL AS Utf8View) LIKEβ |
| | β %google% β |
| | βββββββββββββββββββββββββββββ |
| | |
+---------------+------------------------------------------------------------+
1 row(s) fetched.
Elapsed 0.083 seconds.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
xudong963 commented on issue #15343: URL: https://github.com/apache/datafusion/issues/15343#issuecomment-2742191169 +1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org For queries about this service, please contact Infrastructure at: users@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org For additional commands, e-mail: github-help@datafusion.apache.org
Standing-Man commented on issue #15343: URL: https://github.com/apache/datafusion/issues/15343#issuecomment-2744833718 I agree to use the tree explanation format as the default display, as it is more concise. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org For queries about this service, please contact Infrastructure at: users@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org For additional commands, e-mail: github-help@datafusion.apache.org
irenjj commented on issue #15343: URL: https://github.com/apache/datafusion/issues/15343#issuecomment-2742085140 That's cool, `tree` makes EXPLAIN very concise, and we can easily view detailed information using `EXPLAIN FORMAT xxx`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org For queries about this service, please contact Infrastructure at: users@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org For additional commands, e-mail: github-help@datafusion.apache.org