You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Pranay Singh (Code Review)" <ge...@cloudera.org> on 2018/11/21 04:46:45 UTC
[Impala-ASF-CR] IMPALA-5973: Provide query plan in JSON format
Pranay Singh has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11974
Change subject: IMPALA-5973: Provide query plan in JSON format
......................................................................
IMPALA-5973: Provide query plan in JSON format
EXPLAIN statement provides a query plan in text format, these
changes provide an ability to print the query plan in a JSON format.
The default option to print the query plan is in text format.
In order to print the query plan in JSON format one needs to set
the query option EXPLAIN_FORMAT_JSON to true followed by EXPLAIN
statement to print the query plan.
>set EXPLAIN_FORMAT_JSON=true;
>explain select * from tmp_tbl;
Change-Id: I0f07e2223be58b7323e1ea2fa44b62626cdf4944
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaInternalService.thrift
M common/thrift/ImpalaService.thrift
M fe/src/main/java/org/apache/impala/planner/AggregationNode.java
M fe/src/main/java/org/apache/impala/planner/AnalyticEvalNode.java
M fe/src/main/java/org/apache/impala/planner/CardinalityCheckNode.java
M fe/src/main/java/org/apache/impala/planner/DataSink.java
M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java
M fe/src/main/java/org/apache/impala/planner/DataStreamSink.java
M fe/src/main/java/org/apache/impala/planner/EmptySetNode.java
M fe/src/main/java/org/apache/impala/planner/ExchangeNode.java
M fe/src/main/java/org/apache/impala/planner/HBaseScanNode.java
M fe/src/main/java/org/apache/impala/planner/HBaseTableSink.java
M fe/src/main/java/org/apache/impala/planner/HashJoinNode.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
M fe/src/main/java/org/apache/impala/planner/JoinBuildSink.java
M fe/src/main/java/org/apache/impala/planner/KuduScanNode.java
M fe/src/main/java/org/apache/impala/planner/KuduTableSink.java
M fe/src/main/java/org/apache/impala/planner/NestedLoopJoinNode.java
M fe/src/main/java/org/apache/impala/planner/PlanFragment.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/planner/PlanRootSink.java
M fe/src/main/java/org/apache/impala/planner/Planner.java
M fe/src/main/java/org/apache/impala/planner/SelectNode.java
M fe/src/main/java/org/apache/impala/planner/SingularRowSrcNode.java
M fe/src/main/java/org/apache/impala/planner/SortNode.java
M fe/src/main/java/org/apache/impala/planner/SubplanNode.java
M fe/src/main/java/org/apache/impala/planner/UnionNode.java
M fe/src/main/java/org/apache/impala/planner/UnnestNode.java
31 files changed, 323 insertions(+), 78 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/74/11974/1
--
To view, visit http://gerrit.cloudera.org:8080/11974
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I0f07e2223be58b7323e1ea2fa44b62626cdf4944
Gerrit-Change-Number: 11974
Gerrit-PatchSet: 1
Gerrit-Owner: Pranay Singh (320)
[Impala-ASF-CR] IMPALA-5973: Provide query plan in JSON format
Posted by "Pranay Singh (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/11974
to look at the new patch set (#3).
Change subject: IMPALA-5973: Provide query plan in JSON format
......................................................................
IMPALA-5973: Provide query plan in JSON format
EXPLAIN statement provides a query plan in text format, these
changes provide an ability to print the query plan in a JSON format.
The default option to print the query plan is in text format.
In order to print the query plan in JSON format one needs to set
the query option EXPLAIN_FORMAT_JSON to true followed by EXPLAIN
statement to print the query plan.
>set EXPLAIN_FORMAT_JSON=true;
>explain select * from tmp_tbl;
Change-Id: I0f07e2223be58b7323e1ea2fa44b62626cdf4944
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaInternalService.thrift
M common/thrift/ImpalaService.thrift
M fe/src/main/java/org/apache/impala/planner/AggregationNode.java
M fe/src/main/java/org/apache/impala/planner/AnalyticEvalNode.java
M fe/src/main/java/org/apache/impala/planner/CardinalityCheckNode.java
M fe/src/main/java/org/apache/impala/planner/DataSink.java
M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java
M fe/src/main/java/org/apache/impala/planner/DataStreamSink.java
M fe/src/main/java/org/apache/impala/planner/EmptySetNode.java
M fe/src/main/java/org/apache/impala/planner/ExchangeNode.java
M fe/src/main/java/org/apache/impala/planner/HBaseScanNode.java
M fe/src/main/java/org/apache/impala/planner/HBaseTableSink.java
M fe/src/main/java/org/apache/impala/planner/HashJoinNode.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
M fe/src/main/java/org/apache/impala/planner/JoinBuildSink.java
M fe/src/main/java/org/apache/impala/planner/KuduScanNode.java
M fe/src/main/java/org/apache/impala/planner/KuduTableSink.java
M fe/src/main/java/org/apache/impala/planner/NestedLoopJoinNode.java
M fe/src/main/java/org/apache/impala/planner/PlanFragment.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/planner/PlanRootSink.java
M fe/src/main/java/org/apache/impala/planner/Planner.java
M fe/src/main/java/org/apache/impala/planner/SelectNode.java
M fe/src/main/java/org/apache/impala/planner/SingularRowSrcNode.java
M fe/src/main/java/org/apache/impala/planner/SortNode.java
M fe/src/main/java/org/apache/impala/planner/SubplanNode.java
M fe/src/main/java/org/apache/impala/planner/UnionNode.java
M fe/src/main/java/org/apache/impala/planner/UnnestNode.java
31 files changed, 322 insertions(+), 78 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/74/11974/3
--
To view, visit http://gerrit.cloudera.org:8080/11974
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0f07e2223be58b7323e1ea2fa44b62626cdf4944
Gerrit-Change-Number: 11974
Gerrit-PatchSet: 3
Gerrit-Owner: Pranay Singh (320)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-5973: Provide query plan in JSON format
Posted by "Paul Rogers (Code Review)" <ge...@cloudera.org>.
Paul Rogers has posted comments on this change. ( http://gerrit.cloudera.org:8080/11974 )
Change subject: IMPALA-5973: Provide query plan in JSON format
......................................................................
Patch Set 3:
Hi Pranay, I went ahead and implemented the to-JSON mechanism for the AST: the tree that goes into the planner. I think you can use the same mechanism to generate JSON for the plan that comes out of the planner. Contact me directly if you want details. Or, I'll do a patch in a week or so with the AST work that you can leverage.
--
To view, visit http://gerrit.cloudera.org:8080/11974
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0f07e2223be58b7323e1ea2fa44b62626cdf4944
Gerrit-Change-Number: 11974
Gerrit-PatchSet: 3
Gerrit-Owner: Pranay Singh (320)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pa...@yahoo.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Pranay Singh (320)
Gerrit-Comment-Date: Sat, 08 Dec 2018 03:24:23 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5973: Provide query plan in JSON format
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11974 )
Change subject: IMPALA-5973: Provide query plan in JSON format
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/1420/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/11974
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0f07e2223be58b7323e1ea2fa44b62626cdf4944
Gerrit-Change-Number: 11974
Gerrit-PatchSet: 3
Gerrit-Owner: Pranay Singh (320)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 21 Nov 2018 05:42:05 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5973: Provide query plan in JSON format
Posted by "Pranay Singh (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/11974
to look at the new patch set (#2).
Change subject: IMPALA-5973: Provide query plan in JSON format
......................................................................
IMPALA-5973: Provide query plan in JSON format
EXPLAIN statement provides a query plan in text format, these
changes provide an ability to print the query plan in a JSON format.
The default option to print the query plan is in text format.
In order to print the query plan in JSON format one needs to set
the query option EXPLAIN_FORMAT_JSON to true followed by EXPLAIN
statement to print the query plan.
>set EXPLAIN_FORMAT_JSON=true;
>explain select * from tmp_tbl;
Change-Id: I0f07e2223be58b7323e1ea2fa44b62626cdf4944
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaInternalService.thrift
M common/thrift/ImpalaService.thrift
M fe/src/main/java/org/apache/impala/planner/AggregationNode.java
M fe/src/main/java/org/apache/impala/planner/AnalyticEvalNode.java
M fe/src/main/java/org/apache/impala/planner/CardinalityCheckNode.java
M fe/src/main/java/org/apache/impala/planner/DataSink.java
M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java
M fe/src/main/java/org/apache/impala/planner/DataStreamSink.java
M fe/src/main/java/org/apache/impala/planner/EmptySetNode.java
M fe/src/main/java/org/apache/impala/planner/ExchangeNode.java
M fe/src/main/java/org/apache/impala/planner/HBaseScanNode.java
M fe/src/main/java/org/apache/impala/planner/HBaseTableSink.java
M fe/src/main/java/org/apache/impala/planner/HashJoinNode.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
M fe/src/main/java/org/apache/impala/planner/JoinBuildSink.java
M fe/src/main/java/org/apache/impala/planner/KuduScanNode.java
M fe/src/main/java/org/apache/impala/planner/KuduTableSink.java
M fe/src/main/java/org/apache/impala/planner/NestedLoopJoinNode.java
M fe/src/main/java/org/apache/impala/planner/PlanFragment.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/planner/PlanRootSink.java
M fe/src/main/java/org/apache/impala/planner/Planner.java
M fe/src/main/java/org/apache/impala/planner/SelectNode.java
M fe/src/main/java/org/apache/impala/planner/SingularRowSrcNode.java
M fe/src/main/java/org/apache/impala/planner/SortNode.java
M fe/src/main/java/org/apache/impala/planner/SubplanNode.java
M fe/src/main/java/org/apache/impala/planner/UnionNode.java
M fe/src/main/java/org/apache/impala/planner/UnnestNode.java
31 files changed, 323 insertions(+), 78 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/74/11974/2
--
To view, visit http://gerrit.cloudera.org:8080/11974
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0f07e2223be58b7323e1ea2fa44b62626cdf4944
Gerrit-Change-Number: 11974
Gerrit-PatchSet: 2
Gerrit-Owner: Pranay Singh (320)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-5973: Provide query plan in JSON format
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has abandoned this change. ( http://gerrit.cloudera.org:8080/11974 )
Change subject: IMPALA-5973: Provide query plan in JSON format
......................................................................
Abandoned
I think this is dead for now.
--
To view, visit http://gerrit.cloudera.org:8080/11974
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: abandon
Gerrit-Change-Id: I0f07e2223be58b7323e1ea2fa44b62626cdf4944
Gerrit-Change-Number: 11974
Gerrit-PatchSet: 3
Gerrit-Owner: Pranay Singh (320)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jiawei Wang <ji...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Pranay Singh (320)
[Impala-ASF-CR] IMPALA-5973: Provide query plan in JSON format
Posted by "Paul Rogers (Code Review)" <ge...@cloudera.org>.
Paul Rogers has posted comments on this change. ( http://gerrit.cloudera.org:8080/11974 )
Change subject: IMPALA-5973: Provide query plan in JSON format
......................................................................
Patch Set 3:
Update on this one. Been working on a tricky planner bug. Ended up using the JSON serialization framework in #12079 to create a basic JSON serialization for the plan tree.
Suggestion: to continue with this patch, maybe start with the 12079 serializer and the skeletal work mentioned above. Expand it to all nodes.
The work I did was rough and ready: field names were chosen quickly and I serialized only what I needed. Thought is needed to figure out what to serialize and what to call it.
If we serialize too close to the code, then the JSON form of the plan changes when we make trivial code changes. If we are too abstract, we loose vital details (the current EXPLAIN text, for example, omits estimated cardinality, which is rather vital.)
So there would be huge value in working out the desired JSON structure, implementing serialization for all plan nodes, and wiring it up the the EXPLAIN implementation.
--
To view, visit http://gerrit.cloudera.org:8080/11974
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0f07e2223be58b7323e1ea2fa44b62626cdf4944
Gerrit-Change-Number: 11974
Gerrit-PatchSet: 3
Gerrit-Owner: Pranay Singh (320)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Pranay Singh (320)
Gerrit-Comment-Date: Thu, 27 Dec 2018 21:21:19 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5973: Provide query plan in JSON format
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11974 )
Change subject: IMPALA-5973: Provide query plan in JSON format
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/11974/1/fe/src/main/java/org/apache/impala/planner/PlanFragment.java
File fe/src/main/java/org/apache/impala/planner/PlanFragment.java:
http://gerrit.cloudera.org:8080/#/c/11974/1/fe/src/main/java/org/apache/impala/planner/PlanFragment.java@449
PS1, Line 449: resourceProfile_.multiply(getNumInstancesPerHost(mt_dop)).getExplainString());
line has trailing whitespace
--
To view, visit http://gerrit.cloudera.org:8080/11974
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0f07e2223be58b7323e1ea2fa44b62626cdf4944
Gerrit-Change-Number: 11974
Gerrit-PatchSet: 1
Gerrit-Owner: Pranay Singh (320)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 21 Nov 2018 04:47:46 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5973: Provide query plan in JSON format
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11974 )
Change subject: IMPALA-5973: Provide query plan in JSON format
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/1418/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/11974
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0f07e2223be58b7323e1ea2fa44b62626cdf4944
Gerrit-Change-Number: 11974
Gerrit-PatchSet: 1
Gerrit-Owner: Pranay Singh (320)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 21 Nov 2018 05:22:14 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5973: Provide query plan in JSON format
Posted by "Paul Rogers (Code Review)" <ge...@cloudera.org>.
Paul Rogers has posted comments on this change. ( http://gerrit.cloudera.org:8080/11974 )
Change subject: IMPALA-5973: Provide query plan in JSON format
......................................................................
Patch Set 3:
Pranay, turns out it would be very handy indeed to serialize the query tree to JSON for testing. Your change focuses on the plan tree after planning. My need is for the query tree after analysis. Let's coordinate to see if we can use the same JSON serialization mechanism for both.
--
To view, visit http://gerrit.cloudera.org:8080/11974
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0f07e2223be58b7323e1ea2fa44b62626cdf4944
Gerrit-Change-Number: 11974
Gerrit-PatchSet: 3
Gerrit-Owner: Pranay Singh (320)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pa...@yahoo.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Pranay Singh (320)
Gerrit-Comment-Date: Wed, 05 Dec 2018 21:44:21 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5973: Provide query plan in JSON format
Posted by "Philip Zeyliger (Code Review)" <ge...@cloudera.org>.
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/11974 )
Change subject: IMPALA-5973: Provide query plan in JSON format
......................................................................
Patch Set 3:
It's a little obscure, but, for running queries, we already have a JSON representation of a plan that can be queried using the Impala web server. It's interesting in that it's a representation not just of the plan, but also of some of the execution statistics of said plan. The code for it is here:
// Helper for PlanToJson(), processes a single list of plan nodes which are the
// DFS-flattened representation of a single plan fragment. Called recursively, the
// iterator parameter is updated in place so that when a recursive call returns, the
// caller is pointing at the next of its children.
void PlanToJsonHelper(const map<TPlanNodeId, TPlanNodeExecSummary>& summaries,
const vector<TPlanNode>& nodes,
I think it'd be pretty reasonable to always spit this JSON output into the profile at query completion.
--
To view, visit http://gerrit.cloudera.org:8080/11974
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0f07e2223be58b7323e1ea2fa44b62626cdf4944
Gerrit-Change-Number: 11974
Gerrit-PatchSet: 3
Gerrit-Owner: Pranay Singh (320)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Tue, 27 Nov 2018 19:16:26 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5973: Provide query plan in JSON format
Posted by "Paul Rogers (Code Review)" <ge...@cloudera.org>.
Paul Rogers has posted comments on this change. ( http://gerrit.cloudera.org:8080/11974 )
Change subject: IMPALA-5973: Provide query plan in JSON format
......................................................................
Patch Set 3:
Thanks much for the contribution.
The implementation here is helpful, and is a JSON representation of the explain string, which is a bit different than a JSON representation of the plan itself. The representation here would still require a tool to parse the strings, though now the strings are a bit easier to find.
Since we'd need a parser anyway, we could just parse the original string form. (I wrote such a parser, tedious, but not difficult.)
What we'd like is something that stores each interesting value as a separate JSON key/value pair. Just making something up:
node: {
type: "HDFS-SCAN",
id: "5",
table: "functional.alltypes",
cardinality: 1234,
...
}
To get such a format, we'd have to create a new method on each plan object; we would not want to just overload the existing describe method with additional functionality. This would be quite a bit of work, and we'd have to decide on exactly what we want in the JSON output.
FWIW, if we want the entire plan, in all its gory detail, we can use Thrift:
...
TExecRequest execRequest = frontend_.createExecRequest(queryCtx, explainBuilder);
TSerializer serializer = new TSerializer(new TSimpleJSONProtocol.Factory());
String json = serializer.toString(execRequest);
See: https://stackoverflow.com/questions/21597271/convert-an-object-to-a-json-string-with-thrift-json-serialization
The resulting JSON includes all the details of the plan: it is very large. Run it through here to turn it into a nicely formatted tree: https://jsonformatter.curiousconcept.com
Even if we offer a simplified JSON plan, seems we should also offer the full plan in JSON as the most detailed option.
Or, perhaps we can use the Thrift tree as the basis, with some simplifications to make the JSON more compact.
--
To view, visit http://gerrit.cloudera.org:8080/11974
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0f07e2223be58b7323e1ea2fa44b62626cdf4944
Gerrit-Change-Number: 11974
Gerrit-PatchSet: 3
Gerrit-Owner: Pranay Singh (320)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pa...@yahoo.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Sat, 01 Dec 2018 01:08:31 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5973: Provide query plan in JSON format
Posted by "Jiawei Wang (Code Review)" <ge...@cloudera.org>.
Jiawei Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/11974 )
Change subject: IMPALA-5973: Provide query plan in JSON format
......................................................................
Patch Set 3:
Hi,
I am trying to pick this up. Is there anyone who is familiar with this ticket and its process knows where should I start?
Thanks!
--
To view, visit http://gerrit.cloudera.org:8080/11974
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0f07e2223be58b7323e1ea2fa44b62626cdf4944
Gerrit-Change-Number: 11974
Gerrit-PatchSet: 3
Gerrit-Owner: Pranay Singh (320)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jiawei Wang <ji...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Pranay Singh (320)
Gerrit-Comment-Date: Mon, 08 Jul 2019 21:35:04 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5973: Provide query plan in JSON format
Posted by "Philip Zeyliger (Code Review)" <ge...@cloudera.org>.
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/11974 )
Change subject: IMPALA-5973: Provide query plan in JSON format
......................................................................
Patch Set 3:
This is a continuation of https://gerrit.cloudera.org/#/c/9614/ . I'd prefer it if you re-used the Change-Id, so that we could preserve the previous comments, which still apply.
I took a very very quick pass, and failed to find any examples of what this produces. I want to see some plans side by side to see what we're producing.
My concern with the approach is the same as the previous review:
> If the goal is machine-readability, the approach of hand-crafting JSON via maps seems fragile. I claim that if you want machine-readability, you need to provide the schema of your output, with a way of evolving that schema (e.g., a version number).
That is, JSON is an API, and we need to define the schema of that API explicitly. We could do this by re-using the Thrift, creating a new Thrift, using one of the several JSON schema things (https://json-schema.org/ looks like it's leading the pack, but I'm not sure), or something similar.
--
To view, visit http://gerrit.cloudera.org:8080/11974
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0f07e2223be58b7323e1ea2fa44b62626cdf4944
Gerrit-Change-Number: 11974
Gerrit-PatchSet: 3
Gerrit-Owner: Pranay Singh (320)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Mon, 26 Nov 2018 17:59:24 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5973: Provide query plan in JSON format
Posted by "Pranay Singh (Code Review)" <ge...@cloudera.org>.
Pranay Singh has posted comments on this change. ( http://gerrit.cloudera.org:8080/11974 )
Change subject: IMPALA-5973: Provide query plan in JSON format
......................................................................
Patch Set 3:
Thanks Phil, Paul for your comments will implement it using the Thrift format.
--
To view, visit http://gerrit.cloudera.org:8080/11974
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0f07e2223be58b7323e1ea2fa44b62626cdf4944
Gerrit-Change-Number: 11974
Gerrit-PatchSet: 3
Gerrit-Owner: Pranay Singh (320)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pa...@yahoo.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Pranay Singh (320)
Gerrit-Comment-Date: Sat, 01 Dec 2018 03:15:41 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5973: Provide query plan in JSON format
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11974 )
Change subject: IMPALA-5973: Provide query plan in JSON format
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/1419/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/11974
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0f07e2223be58b7323e1ea2fa44b62626cdf4944
Gerrit-Change-Number: 11974
Gerrit-PatchSet: 2
Gerrit-Owner: Pranay Singh (320)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 21 Nov 2018 05:32:48 +0000
Gerrit-HasComments: No