You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Zoltan Haindrich <ki...@rxd.hu> on 2017/11/28 15:57:45 UTC
Review Request 64122: HIVE-18163 Stats: create materialized view
should also collect stats
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64122/
-----------------------------------------------------------
Review request for hive, Ashutosh Chauhan and Jesús Camacho Rodríguez.
Bugs: HIVE-18163
https://issues.apache.org/jira/browse/HIVE-18163
Repository: hive-git
Description
-------
* collect stats for {{create materialized view}} as well; AFAIK its not possible to do an update on a materialized view ; so every materialized view operation can be considered as a rewrite w.r.t stats
* added a small collection to delay the construction of the view objects in `MaterializedViewRegistry`; the reason this was needed is:
* `StatsTask` runs after `DDLTask` :
* `DDLTask` invoked `MaterializedViewRegistry` to put the view in cache
* `StatsTask` filled out the basicStats info in the metastore...
* next query used the *out-dated* cached table object (which was available at the time {{MaterializedViewRegistry}} built the scanner.
* I've rerun all the "materialized_view" tests. and the results look good to me
* in materialized_view_create_rewrite_2.q.out the usage of {{cmv_mat_view_5}} appeared; which looks good to me
Diffs
-----
ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMaterializedViewsRegistry.java 51b6ef58fc196be716c4b07287fbc83503d1df50
ql/src/java/org/apache/hadoop/hive/ql/plan/BasicStatsWork.java 26bb3e17074bf03cb0fc67c3983252998ab23d4d
ql/src/test/queries/clientpositive/materialized_view_create.q bb50dbb6a1816005873ea51ae193a45d418c99e4
ql/src/test/results/clientpositive/llap/materialized_view_create.q.out 928618390d2ea0254839013f46b8be28a2ff5a54
ql/src/test/results/clientpositive/llap/materialized_view_create_rewrite.q.out 8bebab4ef036b4750da9e00cccf0d0f4ed9c53e8
ql/src/test/results/clientpositive/llap/materialized_view_create_rewrite_2.q.out 83ab7429e40ff9a3313430917e64a1991c47c4df
ql/src/test/results/clientpositive/llap/materialized_view_create_rewrite_multi_db.q.out e1357853631ab369bb79a17911c2e5df0b1e9ac7
ql/src/test/results/clientpositive/llap/materialized_view_describe.q.out 2be1536453843d10644948f6ac7100088ee52e5c
ql/src/test/results/clientpositive/materialized_view_create.q.out 928618390d2ea0254839013f46b8be28a2ff5a54
ql/src/test/results/clientpositive/materialized_view_create_rewrite.q.out aa3240cad4da90b2146e330884e7223708ed20a3
ql/src/test/results/clientpositive/materialized_view_create_rewrite_2.q.out c4bee9c63d4e15181a00af5c7c9e6dbbec443fd9
ql/src/test/results/clientpositive/materialized_view_create_rewrite_3.q.out 9fd70b69371158003284bc55d5965c129bf412a5
ql/src/test/results/clientpositive/materialized_view_create_rewrite_multi_db.q.out a6d00db0f76e0b61ba03b7534cbd8d51dedf5381
ql/src/test/results/clientpositive/materialized_view_describe.q.out 2be1536453843d10644948f6ac7100088ee52e5c
Diff: https://reviews.apache.org/r/64122/diff/1/
Testing
-------
Thanks,
Zoltan Haindrich
Re: Review Request 64122: HIVE-18163 Stats: create materialized view
should also collect stats
Posted by Zoltan Haindrich <ki...@rxd.hu>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64122/
-----------------------------------------------------------
(Updated Nov. 29, 2017, 1:13 p.m.)
Review request for hive, Ashutosh Chauhan and Jesús Camacho Rodríguez.
Changes
-------
update to patch#2
Bugs: HIVE-18163
https://issues.apache.org/jira/browse/HIVE-18163
Repository: hive-git
Description
-------
* collect stats for {{create materialized view}} as well; AFAIK its not possible to do an update on a materialized view ; so every materialized view operation can be considered as a rewrite w.r.t stats
* added a small collection to delay the construction of the view objects in `MaterializedViewRegistry`; the reason this was needed is:
* `StatsTask` runs after `DDLTask` :
* `DDLTask` invoked `MaterializedViewRegistry` to put the view in cache
* `StatsTask` filled out the basicStats info in the metastore...
* next query used the *out-dated* cached table object (which was available at the time {{MaterializedViewRegistry}} built the scanner.
* I've rerun all the "materialized_view" tests. and the results look good to me
* in materialized_view_create_rewrite_2.q.out the usage of {{cmv_mat_view_5}} appeared; which looks good to me
Diffs (updated)
-----
ql/src/java/org/apache/hadoop/hive/ql/QueryLifeTimeHookRunner.java 85e038ce36420859c5e42e081b88065cdad811f5
ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 17640f3396678cec2732bb29033eecd6e8d8db71
ql/src/java/org/apache/hadoop/hive/ql/hooks/MaterializedViewRegistryUpdateHook.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMaterializedViewsRegistry.java 51b6ef58fc196be716c4b07287fbc83503d1df50
ql/src/java/org/apache/hadoop/hive/ql/plan/BasicStatsWork.java 26bb3e17074bf03cb0fc67c3983252998ab23d4d
ql/src/test/queries/clientpositive/materialized_view_create.q bb50dbb6a1816005873ea51ae193a45d418c99e4
ql/src/test/results/clientpositive/beeline/materialized_view_create_rewrite.q.out aa3240cad4da90b2146e330884e7223708ed20a3
ql/src/test/results/clientpositive/llap/materialized_view_create.q.out 928618390d2ea0254839013f46b8be28a2ff5a54
ql/src/test/results/clientpositive/llap/materialized_view_create_rewrite.q.out 8bebab4ef036b4750da9e00cccf0d0f4ed9c53e8
ql/src/test/results/clientpositive/llap/materialized_view_create_rewrite_2.q.out 83ab7429e40ff9a3313430917e64a1991c47c4df
ql/src/test/results/clientpositive/llap/materialized_view_create_rewrite_multi_db.q.out e1357853631ab369bb79a17911c2e5df0b1e9ac7
ql/src/test/results/clientpositive/llap/materialized_view_describe.q.out 2be1536453843d10644948f6ac7100088ee52e5c
ql/src/test/results/clientpositive/materialized_view_create.q.out 928618390d2ea0254839013f46b8be28a2ff5a54
ql/src/test/results/clientpositive/materialized_view_create_rewrite.q.out aa3240cad4da90b2146e330884e7223708ed20a3
ql/src/test/results/clientpositive/materialized_view_create_rewrite_2.q.out c4bee9c63d4e15181a00af5c7c9e6dbbec443fd9
ql/src/test/results/clientpositive/materialized_view_create_rewrite_3.q.out 9fd70b69371158003284bc55d5965c129bf412a5
ql/src/test/results/clientpositive/materialized_view_create_rewrite_multi_db.q.out a6d00db0f76e0b61ba03b7534cbd8d51dedf5381
ql/src/test/results/clientpositive/materialized_view_describe.q.out 2be1536453843d10644948f6ac7100088ee52e5c
Diff: https://reviews.apache.org/r/64122/diff/2/
Changes: https://reviews.apache.org/r/64122/diff/1-2/
Testing
-------
Thanks,
Zoltan Haindrich