You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "abhioncbr (via GitHub)" <gi...@apache.org> on 2023/07/06 14:41:33 UTC
[GitHub] [pinot] abhioncbr opened a new issue, #11039: Support different output formats(XML, JSON, DOT and TEXT)) for explain physical plan
abhioncbr opened a new issue, #11039:
URL: https://github.com/apache/pinot/issues/11039
**Problem**
Currently, `ExplainPlanPlanVisitor.explain` ([code reference](https://github.com/apache/pinot/blob/master/pinot-query-planner/src/main/java/org/apache/pinot/query/planner/ExplainPlanPlanVisitor.java#L63)) provides the physical plan of the SQL query in a simple text format. One such example of the output is as follows.
```
[0]@localhost:3 MAIL_RECEIVE(RANDOM_DISTRIBUTED)
├── [1]@localhost:2 MAIL_SEND(RANDOM_DISTRIBUTED)->{[0]@localhost@{3,3}|[0]}
│ └── [1]@localhost:2 PROJECT
│ └── [1]@localhost:2 TABLE SCAN (a) null
└── [1]@localhost:1 MAIL_SEND(RANDOM_DISTRIBUTED)->{[0]@localhost@{3,3}|[0]}
└── [1]@localhost:1 PROJECT
└── [1]@localhost:1 TABLE SCAN (a) null
```
`EXPLAIN PLAN FOR` SQL query supports various formats(XML, JSON, DOT and TEXT), and with the solution approach to the [issue](https://github.com/apache/pinot/issues/10901)(support explain on physical plan ); the output of the physical plan of the query should follow the same format as mentioned in the `Explain` Query.
**Solution Approach**
- Extend the `ExplainPlanPlanVisitor` class for different output formats and generate the physical plan based on the format mentioned in the `Explain` query.
- `TEXT` will be the default output format, if not mentioned in the `Explain` query.
**NOTE**
example of the `Explain` query with format option
```
EXPLAIN Plan As JSON Select * from airlineStats
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] [multistage]: Support different output formats(XML, JSON, DOT and TEXT) for explain physical plan [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on issue #11039:
URL: https://github.com/apache/pinot/issues/11039#issuecomment-2014605480
I've created [this document](https://docs.google.com/document/d/14YD-Tbb2NxxMULwpFX4b6uC9YjhygPYgiHvI0jt0q0s/edit#heading=h.om79hm58o58) to try to summarize the current explain options we want and to propose a unified future version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] [multistage]: Support different output formats(XML, JSON, DOT and TEXT) for explain physical plan [pinot]
Posted by "abhioncbr (via GitHub)" <gi...@apache.org>.
abhioncbr commented on issue #11039:
URL: https://github.com/apache/pinot/issues/11039#issuecomment-2012131763
Referring the [PR](https://github.com/apache/pinot/pull/11052) where we have introduced this.
@gortiz mentioned it correctly, We want the physical and logical plan separately. If I remember it correctly, following the standardized Calcite approach, we weren't able to to get the physical plan alone. Logical plan is always there and hence we came up with the Pinot specific way to get physical plan.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] walterddr commented on issue #11039: [multistage]: Support different output formats(XML, JSON, DOT and TEXT) for explain physical plan
Posted by "walterddr (via GitHub)" <gi...@apache.org>.
walterddr commented on issue #11039:
URL: https://github.com/apache/pinot/issues/11039#issuecomment-1627531561
i think it is fine not to do the formatting for physical plan. it is no longer a tree (but a DAG) so
- it doesn't seem desirable to support the JSON and XML format which technically still is a tree structure
- supporting DOT is definitely desirable but i would assume the effort would be a lot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] [multistage]: Support different output formats(XML, JSON, DOT and TEXT) for explain physical plan [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on issue #11039:
URL: https://github.com/apache/pinot/issues/11039#issuecomment-2006736803
I think we are merging terms here. Our custom physical implementation may be a DAG, but the plan is always a tree.
At the same time it is not clear to me the difference between Calcite default `EXPLAIN PLAN WITH IMPLEMENTATION` and our extension `EXPLAIN IMPLEMENTATION PLAN`. But having the two versions is very confusing and right now (1.1.0) `EXPLAIN PLAN WITH IMPLEMENTATION` does literally does the same thing as `EXPLAIN PLAN`, which is even more confusing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] [multistage]: Support different output formats(XML, JSON, DOT and TEXT) for explain physical plan [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on issue #11039:
URL: https://github.com/apache/pinot/issues/11039#issuecomment-2011771245
I was studying the situation. In Calcite explain can be decorated with:
- `AS <TEXT, XML, JSON, DOT>` to change the format used to print the result
- `WITH IMPLEMENTATION`, `WITHOUT IMPLEMENTATION`, `WITH TYPE` to ask for the logical (WITHOUT IMPLEMENTATION) or physical (WITH IMPLEMENTATION) plan. I'm not sure about the meaning of `WITH TYPE`
- `INCLUDING ATTRIBUTES`, `EXCLUDING ATTRIBUTES`. I'm not sure, but I guess it changes which attributes are included on each node (rows, time, etc)
In multi-stage we honor `AS` but neither `WITH` or `INCLUDING`. I think there are two reasons why we don't follow `WITH` semantic:
1. Right now it is very difficult to generate a physical plan using Calcite objects
2. `WITH IMPLEMENTATION` is enabled by default in Calcite, so the SqlExplain instance we receive when user writes `EXPLAIN PLAN FOR select ...` returns true to `withImplementation()`. That means that by default customers would receive the physical plan, not the logical like we are doing right now.
I think we should keep this situation for now and work to try to be able to have the actual physical plan (including used indexes). Once we have that, we can return that by default with `EXPLAIN` and optionally show the logical plan when customers ask for that (using `WITHOUT IMPLEMENTATION`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] abhioncbr commented on issue #11039: [multistage]: Support different output formats(XML, JSON, DOT and TEXT) for explain physical plan
Posted by "abhioncbr (via GitHub)" <gi...@apache.org>.
abhioncbr commented on issue #11039:
URL: https://github.com/apache/pinot/issues/11039#issuecomment-1627553179
Okay, I got the point. Let me close the issue and in future, if we want to support `DOT`, we will create a new task. Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] [multistage]: Support different output formats(XML, JSON, DOT and TEXT) for explain physical plan [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #11039:
URL: https://github.com/apache/pinot/issues/11039#issuecomment-2010936793
@gortiz Seems like `EXPLAIN IMPLEMENTATION PLAN` is Pinot customized SQL syntax. Is `EXPLAIN PLAN WITH IMPLEMENTATION` a commonly used way for showing physical plan? If so, I'd suggest moving to the standard syntax
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] abhioncbr closed issue #11039: [multistage]: Support different output formats(XML, JSON, DOT and TEXT) for explain physical plan
Posted by "abhioncbr (via GitHub)" <gi...@apache.org>.
abhioncbr closed issue #11039: [multistage]: Support different output formats(XML, JSON, DOT and TEXT) for explain physical plan
URL: https://github.com/apache/pinot/issues/11039
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] [multistage]: Support different output formats(XML, JSON, DOT and TEXT) for explain physical plan [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on issue #11039:
URL: https://github.com/apache/pinot/issues/11039#issuecomment-2014622111
I've created https://github.com/apache/pinot/issues/12696 to show the current explain state and to propose a new version where explain behaves in the same way in V1 and V2. I'm not planning to work on this in the short term but I think it may be useful to have it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org