You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Aman Sinha (Code Review)" <ge...@cloudera.org> on 2021/06/15 07:03:06 UTC

[Impala-ASF-CR] IMPALA-10723: [WIP] Treat materialized view as a table instead of a view

Aman Sinha has uploaded this change for review. ( http://gerrit.cloudera.org:8080/17595


Change subject: IMPALA-10723: [WIP] Treat materialized view as a table instead of a view
......................................................................

IMPALA-10723: [WIP] Treat materialized view as a table instead of a view

The existing behavior is that materialized views are treated
as views and therefore expanded similar to a view when one
queries the MV directly (SELECT * FROM materialized_view).
This is incorrect since an MV is a regular table with physical
properties such as partitioning, clustering etc. and should be
treated as such even though it has a view definition associated
with it.

This patch focuses on the use case where MVs are created as
HDFS tables and makes the MVs a derived class of HdfsTable,
therefore making it a Table object. It adds support for
collecting and displaying statistics on materialized views
and these statistics could be leveraged by an external frontend
that supports MV based query rewrites. Directly querying a MV
is permitted but inserts into MVs is not since MVs are supposed
to be only modified through an external refresh when the source
tables have modifications.

If the source tables associated with a materialized view have
column masking or row-filtering Ranger policies, querying the
MV will throw an error. This behavior is consistent with that
of Hive.

Testing
 - Added tests for compute stats, drop stats, show stats and
   simple select query on an MV.
 - Added a simple test for column masking of source table

TODO:
 - The column masking for the source table does not seem to be
   recognized and querying the MV succeeds instead of failing.
   Need to understand why.

Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
---
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java
A fe/src/main/java/org/apache/impala/catalog/MaterializedViewHdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java
M testdata/workloads/functional-planner/queries/PlannerTest/views.test
M testdata/workloads/functional-query/queries/QueryTest/compute-stats.test
9 files changed, 205 insertions(+), 3 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/95/17595/1
-- 
To view, visit http://gerrit.cloudera.org:8080/17595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Gerrit-Change-Number: 17595
Gerrit-PatchSet: 1
Gerrit-Owner: Aman Sinha <am...@cloudera.com>

[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view

Posted by "Aman Sinha (Code Review)" <ge...@cloudera.org>.
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/17595 )

Change subject: IMPALA-10723: Treat materialized view as a table instead of a view
......................................................................


Patch Set 13:

(6 comments)

http://gerrit.cloudera.org:8080/#/c/17595/11/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
File fe/src/main/java/org/apache/impala/analysis/Analyzer.java:

http://gerrit.cloudera.org:8080/#/c/17595/11/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@225
PS11, Line 225: vAuthExceptionM
> It seems the suffix is not added yet. BTW, I think we can remove the boolea
PS 12 was showing the suffix.  In any case, in PS 13, I removed the boolean and am using the null check as suggested.


http://gerrit.cloudera.org:8080/#/c/17595/11/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@931
PS11, Line 931:           resolvedTableRef.getTable() instanceof MaterializedViewHdfsTable &&
> If the MV has masking policies, we won't go to the below check. Should we m
Good point.  I moved the entire check for source table masking up into an else-if block before going into the check for tableMask.needsMaskingOrFiltering().


http://gerrit.cloudera.org:8080/#/c/17595/11/fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java
File fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java:

http://gerrit.cloudera.org:8080/#/c/17595/11/fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java@322
PS11, Line 322:         ((MaterializedViewHdfsTable) tbl).addSrcTables(mvSrcTableNames);
> Can we define MV on some real views? In this case, 'mvSrcTableNames' could 
MVs are not allowed to be defined on regular views since MVs expect the source to be transactional entities and views are not in that category.  So, in Hive, the following fails (ran through beeline):
   0: jdbc:hive2://localhost:11050/default> create view vv1 as select * from tpch.nation;
   0: jdbc:hive2://localhost:11050/default> create materialized view mvv1 as select n_nationkey, count(*) from vv1 group by n_nationkey;
   Error: Error while compiling statement: FAILED: SemanticException Automatic rewriting for materialized view cannot be enabled if the materialized view uses non-transactional tables (state=42000,code=40000)


http://gerrit.cloudera.org:8080/#/c/17595/11/fe/src/main/java/org/apache/impala/catalog/MaterializedViewHdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/MaterializedViewHdfsTable.java:

http://gerrit.cloudera.org:8080/#/c/17595/11/fe/src/main/java/org/apache/impala/catalog/MaterializedViewHdfsTable.java@84
PS11, Line 84:   }
> nit: need a blank line after this
Done


http://gerrit.cloudera.org:8080/#/c/17595/11/fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java
File fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java:

http://gerrit.cloudera.org:8080/#/c/17595/11/fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java@3171
PS11, Line 3171:         onTable("functional", "mv1_alltypes_jointbl", TPrivilegeLevel.SELECT));
> Can we add an additional test that also add a masking policies on the MV? I
Yup, added a column masking policy on mv1_alltypes_jointbl and tested that the same authorization exception is thrown (this after the other change to Analyzer#resolveTableMask()).


http://gerrit.cloudera.org:8080/#/c/17595/7/testdata/workloads/functional-query/queries/QueryTest/compute-stats.test
File testdata/workloads/functional-query/queries/QueryTest/compute-stats.test:

http://gerrit.cloudera.org:8080/#/c/17595/7/testdata/workloads/functional-query/queries/QueryTest/compute-stats.test@1841
PS7, Line 1841: G
> Thanks for the explanation. 
I will explore the syntax extension for the show stats command in a follow-up jira and we can discuss what would be the appropriate syntax.



-- 
To view, visit http://gerrit.cloudera.org:8080/17595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Gerrit-Change-Number: 17595
Gerrit-PatchSet: 13
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 06 Apr 2022 06:40:41 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17595 )

Change subject: IMPALA-10723: Treat materialized view as a table instead of a view
......................................................................


Patch Set 13: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8026/


-- 
To view, visit http://gerrit.cloudera.org:8080/17595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Gerrit-Change-Number: 17595
Gerrit-PatchSet: 13
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 06 Apr 2022 11:08:35 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view

Posted by "Aman Sinha (Code Review)" <ge...@cloudera.org>.
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/17595 )

Change subject: IMPALA-10723: Treat materialized view as a table instead of a view
......................................................................


Patch Set 5:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/17595/4/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
File fe/src/main/java/org/apache/impala/analysis/Analyzer.java:

http://gerrit.cloudera.org:8080/#/c/17595/4/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@892
PS4, Line 892:             .isReferencesMaskedTables(authChecker, getCatalog(), getUser())) {
> tab used for whitespace
Done


http://gerrit.cloudera.org:8080/#/c/17595/3/fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java
File fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java:

http://gerrit.cloudera.org:8080/#/c/17595/3/fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java@628
PS3, Line 628:         "access type is: READONLY");
> tab used for whitespace
Done


http://gerrit.cloudera.org:8080/#/c/17595/3/fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java@637
PS3, Line 637:         "access type is: READONLY");
> tab used for whitespace
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/17595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Gerrit-Change-Number: 17595
Gerrit-PatchSet: 5
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 16 Jun 2021 01:31:30 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view

Posted by "Aman Sinha (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Qifan Chen, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/17595

to look at the new patch set (#10).

Change subject: IMPALA-10723: Treat materialized view as a table instead of a view
......................................................................

IMPALA-10723: Treat materialized view as a table instead of a view

The existing behavior is that materialized views are treated
as views and therefore expanded similar to a view when one
queries the MV directly (SELECT * FROM materialized_view).
This is incorrect since an MV is a regular table with physical
properties such as partitioning, clustering etc. and should be
treated as such even though it has a view definition associated
with it.

This patch focuses on the use case where MVs are created as
HDFS tables and makes the MVs a derived class of HdfsTable,
therefore making it a Table object. It adds support for
collecting and displaying statistics on materialized views
and these statistics could be leveraged by an external frontend
that supports MV based query rewrites (note that such a rewrite
is not supported by Impala with or without this patch).

Directly querying a MV is permitted but inserts into MVs is not
since MVs are supposed to be only modified through an external
refresh when the source tables have modifications.

If the source tables associated with a materialized view have
column masking or row-filtering Ranger policies, querying the
MV will throw an error. This behavior is consistent with that
of Hive.

Testing:
 - Added transactional tables for alltypes, jointbl and used
   them as source tables to create materialized view.
 - Added tests for compute stats, drop stats, show stats and
   simple select query on a materialized view.
 - Added test for select on a materialized view when the
   source table has a column mask.
 - Modified analyzer tests related to alter, insert, drop of
   materialized view.

Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
---
M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java
A fe/src/main/java/org/apache/impala/catalog/MaterializedViewHdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java
M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java
M fe/src/test/java/org/apache/impala/authorization/AuthorizationTestBase.java
M fe/src/test/java/org/apache/impala/service/FrontendTest.java
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
M testdata/workloads/functional-planner/queries/PlannerTest/views.test
M testdata/workloads/functional-query/queries/QueryTest/compute-stats.test
M testdata/workloads/functional-query/queries/QueryTest/views.test
16 files changed, 307 insertions(+), 9 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/95/17595/10
-- 
To view, visit http://gerrit.cloudera.org:8080/17595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Gerrit-Change-Number: 17595
Gerrit-PatchSet: 10
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>

[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view

Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/17595 )

Change subject: IMPALA-10723: Treat materialized view as a table instead of a view
......................................................................


Patch Set 12:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/17595/7/testdata/workloads/functional-query/queries/QueryTest/compute-stats.test
File testdata/workloads/functional-query/queries/QueryTest/compute-stats.test:

http://gerrit.cloudera.org:8080/#/c/17595/7/testdata/workloads/functional-query/queries/QueryTest/compute-stats.test@1841
PS7, Line 1841: G
> Adding the MV syntax is not in scope for this patch because the DDLs for MV
Thanks for the explanation. 

I still feel it is useful to distinguish tables from MVs at SQL level. What HIVE does in DDLs for MVs are right. If we can carry it to the IMPALA commands on MVs (such as show stats in this patch), we maintain that property.



-- 
To view, visit http://gerrit.cloudera.org:8080/17595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Gerrit-Change-Number: 17595
Gerrit-PatchSet: 12
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 24 Mar 2022 13:29:07 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17595 )

Change subject: IMPALA-10723: Treat materialized view as a table instead of a view
......................................................................


Patch Set 12: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7961/


-- 
To view, visit http://gerrit.cloudera.org:8080/17595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Gerrit-Change-Number: 17595
Gerrit-PatchSet: 12
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 24 Mar 2022 06:32:11 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17595 )

Change subject: IMPALA-10723: Treat materialized view as a table instead of a view
......................................................................


Patch Set 12:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/10331/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Gerrit-Change-Number: 17595
Gerrit-PatchSet: 12
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 24 Mar 2022 02:42:13 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17595 )

Change subject: IMPALA-10723: Treat materialized view as a table instead of a view
......................................................................


Patch Set 8:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7948/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/17595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Gerrit-Change-Number: 17595
Gerrit-PatchSet: 8
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Sun, 20 Mar 2022 22:23:47 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/17595 )

Change subject: IMPALA-10723: Treat materialized view as a table instead of a view
......................................................................


Patch Set 15: Code-Review+2

Carry Qifan's +1. Thanks a lot for this feature, Aman!


-- 
To view, visit http://gerrit.cloudera.org:8080/17595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Gerrit-Change-Number: 17595
Gerrit-PatchSet: 15
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 14 Apr 2022 11:56:05 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17595 )

Change subject: IMPALA-10723: Treat materialized view as a table instead of a view
......................................................................


Patch Set 13:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8026/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/17595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Gerrit-Change-Number: 17595
Gerrit-PatchSet: 13
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 06 Apr 2022 06:44:48 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/17595 )

Change subject: IMPALA-10723: Treat materialized view as a table instead of a view
......................................................................


Patch Set 13:

(1 comment)

> Patch Set 13: Verified-1
> 
> Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8026/

I take a look at file-list-begin-1.log in one of the build. There are no files under the functional_parquet.alltypes_transactional and functional_parquet.jointbl_transactional tables. Neither can I find statements loading these two tables in logs/data_loading/sql/functional/load-functional-query-exhaustive-impala-generated-parquet-none-none.sql.

Note that these tables under the functional db is ok (non-empty). However, I can't find the rebuild statement on the MV in logs/data_loading/sql/functional/load-functional-query-exhaustive-hive-generated-text-none-none.sql.

http://gerrit.cloudera.org:8080/#/c/17595/11/fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java
File fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java:

http://gerrit.cloudera.org:8080/#/c/17595/11/fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java@322
PS11, Line 322:         ((MaterializedViewHdfsTable) tbl).addSrcTables(mvSrcTableNames);
> MVs are not allowed to be defined on regular views since MVs expect the sou
I see. Thanks for the explanation!



-- 
To view, visit http://gerrit.cloudera.org:8080/17595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Gerrit-Change-Number: 17595
Gerrit-PatchSet: 13
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 08 Apr 2022 08:07:27 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/17595 )

Change subject: IMPALA-10723: Treat materialized view as a table instead of a view
......................................................................


Patch Set 12:

(6 comments)

> Patch Set 12:
> 
> > Patch Set 12: Verified-1
> > 
> > Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7961/
> 
> A few of the new tests I added for this patch failed due to the materialized view table being empty. On my local machine it works fine.  I looked at the logs from the jenkins run and see that the CREATE MATERIALIZED VIEW .. FROM alltypes_transactional join jointbl_transactional ... is being executed even before the INSERT commands into those source tables are executed, so the MV remains empty.  I will check if this can be addressed through a DEPENDENT_LOAD_HIVE section.  Not sure if this is applicable for materialized views.

I think we can refresh the MV in the DEPENDENT_LOAD_HIVE section:

 ALTER MATERIALIZED VIEW [db_name.]materialized_view_name REBUILD;

https://cwiki.apache.org/confluence/display/Hive/Materialized+views#Materializedviews-Materializedviewmaintenance

BTW, really happy to see this work getting revived. Thanks for working on it!

http://gerrit.cloudera.org:8080/#/c/17595/5//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/17595/5//COMMIT_MSG@18
PS5, Line 18: makes the MVs a derived class of HdfsTable
> Yes, for now this patch ignores Kudu. 
Sure. I think the GVO job will verifiy the ranger authorization part in the local catalog mode. Let's see if it catches any issues.


http://gerrit.cloudera.org:8080/#/c/17595/11/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
File fe/src/main/java/org/apache/impala/analysis/Analyzer.java:

http://gerrit.cloudera.org:8080/#/c/17595/11/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@225
PS11, Line 225: mvAuthException
> added the suffix.
It seems the suffix is not added yet. BTW, I think we can remove the boolean variable and replace it with a check on (mvAuthExceptionMsg != null).


http://gerrit.cloudera.org:8080/#/c/17595/11/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@931
PS11, Line 931:       } else if (tableMask.needsMaskingOrFiltering()) {
If the MV has masking policies, we won't go to the below check. Should we move the added check before this?


http://gerrit.cloudera.org:8080/#/c/17595/11/fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java
File fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java:

http://gerrit.cloudera.org:8080/#/c/17595/11/fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java@322
PS11, Line 322:         ((MaterializedViewHdfsTable) tbl).addSrcTables(mvSrcTableNames);
Can we define MV on some real views? In this case, 'mvSrcTableNames' could be a view name. addSrcTables() doens't add all the source tables.

Note that collectTableCandidates() can only collect the table names. They are not resolved until we go into this method recursively again (at line 337).


http://gerrit.cloudera.org:8080/#/c/17595/11/fe/src/main/java/org/apache/impala/catalog/MaterializedViewHdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/MaterializedViewHdfsTable.java:

http://gerrit.cloudera.org:8080/#/c/17595/11/fe/src/main/java/org/apache/impala/catalog/MaterializedViewHdfsTable.java@84
PS11, Line 84:   }
nit: need a blank line after this


http://gerrit.cloudera.org:8080/#/c/17595/11/fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java
File fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java:

http://gerrit.cloudera.org:8080/#/c/17595/11/fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java@3171
PS11, Line 3171: 
Can we add an additional test that also add a masking policies on the MV? IIUC, we should see the same error.
Just want to add the test coverage for the question I commented in Analyzer#resolveTableMask().



-- 
To view, visit http://gerrit.cloudera.org:8080/17595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Gerrit-Change-Number: 17595
Gerrit-PatchSet: 12
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 24 Mar 2022 08:54:27 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view

Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/17595 )

Change subject: IMPALA-10723: Treat materialized view as a table instead of a view
......................................................................


Patch Set 11:

(7 comments)

Looks great and many thanks for the rework!

http://gerrit.cloudera.org:8080/#/c/17595/11/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
File fe/src/main/java/org/apache/impala/analysis/Analyzer.java:

http://gerrit.cloudera.org:8080/#/c/17595/11/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@225
PS11, Line 225: mvAuthException
nit. missing suffix _ for these two new fields?


http://gerrit.cloudera.org:8080/#/c/17595/7/fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
File fe/src/main/java/org/apache/impala/analysis/InsertStmt.java:

http://gerrit.cloudera.org:8080/#/c/17595/7/fe/src/main/java/org/apache/impala/analysis/InsertStmt.java@464
PS7, Line 464: table_ instanceof MaterializedViewHdfsTable
> Impala only allows updates/deletes on Kudu tables, so if you do an UPDATE o
Done


http://gerrit.cloudera.org:8080/#/c/17595/11/fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java
File fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java:

http://gerrit.cloudera.org:8080/#/c/17595/11/fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java@625
PS11, Line 625: table
MV keyword instead?


http://gerrit.cloudera.org:8080/#/c/17595/11/fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java@635
PS11, Line 635: drop table functional.materialized_view 
Should we replace table with keyword MV here? 

Please refer to my other comment on making a distinction between a table and a MV at SQL level. 

In this doc, https://docs.cloudera.com/runtime/7.2.6/materialized-view-commands/topics/hive_drop_materialized_view.html, drop MV is used.


http://gerrit.cloudera.org:8080/#/c/17595/11/testdata/workloads/functional-planner/queries/PlannerTest/views.test
File testdata/workloads/functional-planner/queries/PlannerTest/views.test:

http://gerrit.cloudera.org:8080/#/c/17595/11/testdata/workloads/functional-planner/queries/PlannerTest/views.test@589
PS11, Line 589: 
May add a test to generate a distributed plan for the above query to exercise the distributed planner code.


http://gerrit.cloudera.org:8080/#/c/17595/7/testdata/workloads/functional-query/queries/QueryTest/compute-stats.test
File testdata/workloads/functional-query/queries/QueryTest/compute-stats.test:

http://gerrit.cloudera.org:8080/#/c/17595/7/testdata/workloads/functional-query/queries/QueryTest/compute-stats.test@1841
PS7, Line 1841: table
> You mean 'show materialized view  stats .. ' ?  Since this is now treated a
Yeah, I am considering this too, as what it means to treat MVs as tables. It seems to me the idea of this patch is to exploit the benefits of MVs being physically stored as tables. But logically, MVs are still MVs and not tables, at the SQL interface level. Therefore one should be able to create table 'foo' and mv 'foo'. 

If we follow this idea, then we need to make the distinction for certain SQL statements, such as SHOW TABLE STATS <table> and SHOW MV STATS <mv>, or SHOW CREATE TABLE <table> and SHOW CREATE MV <mv>.


http://gerrit.cloudera.org:8080/#/c/17595/11/testdata/workloads/functional-query/queries/QueryTest/compute-stats.test
File testdata/workloads/functional-query/queries/QueryTest/compute-stats.test:

http://gerrit.cloudera.org:8080/#/c/17595/11/testdata/workloads/functional-query/queries/QueryTest/compute-stats.test@1841
PS11, Line 1841: table
See my comment on the distinction between table and MV at SQL level.



-- 
To view, visit http://gerrit.cloudera.org:8080/17595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Gerrit-Change-Number: 17595
Gerrit-PatchSet: 11
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 23 Mar 2022 13:11:15 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view

Posted by "Aman Sinha (Code Review)" <ge...@cloudera.org>.
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/17595 )

Change subject: IMPALA-10723: Treat materialized view as a table instead of a view
......................................................................


Patch Set 15:

> Patch Set 14:
> 
> (1 comment)

Coincidentally, that's what I concluded just a little while ago after playing around with the compute stats.  On my local machine I was getting 503B consistently for mv2_alltypes_jointbl.  I have replaced with the regex.


-- 
To view, visit http://gerrit.cloudera.org:8080/17595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Gerrit-Change-Number: 17595
Gerrit-PatchSet: 15
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 14 Apr 2022 05:08:05 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17595 )

Change subject: IMPALA-10723: Treat materialized view as a table instead of a view
......................................................................


Patch Set 13:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/10404/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Gerrit-Change-Number: 17595
Gerrit-PatchSet: 13
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 06 Apr 2022 06:45:14 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17595 )

Change subject: IMPALA-10723: Treat materialized view as a table instead of a view
......................................................................


Patch Set 8:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/10312/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Gerrit-Change-Number: 17595
Gerrit-PatchSet: 8
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Sun, 20 Mar 2022 20:19:57 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view

Posted by "Aman Sinha (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/17595

to look at the new patch set (#5).

Change subject: IMPALA-10723: Treat materialized view as a table instead of a view
......................................................................

IMPALA-10723: Treat materialized view as a table instead of a view

The existing behavior is that materialized views are treated
as views and therefore expanded similar to a view when one
queries the MV directly (SELECT * FROM materialized_view).
This is incorrect since an MV is a regular table with physical
properties such as partitioning, clustering etc. and should be
treated as such even though it has a view definition associated
with it.

This patch focuses on the use case where MVs are created as
HDFS tables and makes the MVs a derived class of HdfsTable,
therefore making it a Table object. It adds support for
collecting and displaying statistics on materialized views
and these statistics could be leveraged by an external frontend
that supports MV based query rewrites. Directly querying a MV
is permitted but inserts into MVs is not since MVs are supposed
to be only modified through an external refresh when the source
tables have modifications.

If the source tables associated with a materialized view have
column masking or row-filtering Ranger policies, querying the
MV will throw an error. This behavior is consistent with that
of Hive.

Testing:
 - Added tests for compute stats, drop stats, show stats and
   simple select query on an MV.
 - Added test for select on a materialized view when the
   source table has a column mask.
 - Modified analyzer tests related to alter, insert, drop of
   materialized view.

Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
---
M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java
A fe/src/main/java/org/apache/impala/catalog/MaterializedViewHdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java
M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java
M fe/src/test/java/org/apache/impala/authorization/AuthorizationTestBase.java
M testdata/workloads/functional-planner/queries/PlannerTest/views.test
M testdata/workloads/functional-query/queries/QueryTest/compute-stats.test
12 files changed, 230 insertions(+), 6 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/95/17595/5
-- 
To view, visit http://gerrit.cloudera.org:8080/17595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Gerrit-Change-Number: 17595
Gerrit-PatchSet: 5
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>

[Impala-ASF-CR] IMPALA-10723: [WIP] Treat materialized view as a table instead of a view

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17595 )

Change subject: IMPALA-10723: [WIP] Treat materialized view as a table instead of a view
......................................................................


Patch Set 2:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/8918/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Gerrit-Change-Number: 17595
Gerrit-PatchSet: 2
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 15 Jun 2021 08:19:44 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17595 )

Change subject: IMPALA-10723: Treat materialized view as a table instead of a view
......................................................................


Patch Set 12:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7961/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/17595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Gerrit-Change-Number: 17595
Gerrit-PatchSet: 12
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 24 Mar 2022 01:43:01 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17595 )

Change subject: IMPALA-10723: Treat materialized view as a table instead of a view
......................................................................


Patch Set 14: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8033/


-- 
To view, visit http://gerrit.cloudera.org:8080/17595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Gerrit-Change-Number: 17595
Gerrit-PatchSet: 14
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Mon, 11 Apr 2022 09:35:47 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view

Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/17595 )

Change subject: IMPALA-10723: Treat materialized view as a table instead of a view
......................................................................


Patch Set 13: Code-Review+1

(1 comment)

Looks good!

http://gerrit.cloudera.org:8080/#/c/17595/7/testdata/workloads/functional-query/queries/QueryTest/compute-stats.test
File testdata/workloads/functional-query/queries/QueryTest/compute-stats.test:

http://gerrit.cloudera.org:8080/#/c/17595/7/testdata/workloads/functional-query/queries/QueryTest/compute-stats.test@1841
PS7, Line 1841: G
> I will explore the syntax extension for the show stats command in a follow-
Sounds like a plan.



-- 
To view, visit http://gerrit.cloudera.org:8080/17595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Gerrit-Change-Number: 17595
Gerrit-PatchSet: 13
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 06 Apr 2022 18:13:45 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/17595 )

Change subject: IMPALA-10723: Treat materialized view as a table instead of a view
......................................................................

IMPALA-10723: Treat materialized view as a table instead of a view

The existing behavior is that materialized views are treated
as views and therefore expanded similar to a view when one
queries the MV directly (SELECT * FROM materialized_view).
This is incorrect since an MV is a regular table with physical
properties such as partitioning, clustering etc. and should be
treated as such even though it has a view definition associated
with it.

This patch focuses on the use case where MVs are created as
HDFS tables and makes the MVs a derived class of HdfsTable,
therefore making it a Table object. It adds support for
collecting and displaying statistics on materialized views
and these statistics could be leveraged by an external frontend
that supports MV based query rewrites (note that such a rewrite
is not supported by Impala with or without this patch). Note
that we are not introducing new syntax for MVs since DDL, DML
operations on MVs are only supported through Hive.

Directly querying a MV is permitted but inserts into MVs is not
since MVs are supposed to be only modified through an external
refresh when the source tables have modifications.

If the source tables associated with a materialized view have
column masking or row-filtering Ranger policies, querying the
MV will throw an error. This behavior is consistent with that
of Hive.

Testing:
 - Added transactional tables for alltypes, jointbl and used
   them as source tables to create materialized view.
 - Added tests for compute stats, drop stats, show stats and
   simple select query on a materialized view.
 - Added test for select on a materialized view when the
   source table has a column mask.
 - Modified analyzer tests related to alter, insert, drop of
   materialized view.

Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Reviewed-on: http://gerrit.cloudera.org:8080/17595
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Quanlong Huang <hu...@gmail.com>
---
M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java
A fe/src/main/java/org/apache/impala/catalog/MaterializedViewHdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java
M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java
M fe/src/test/java/org/apache/impala/authorization/AuthorizationTestBase.java
M testdata/bin/compute-table-stats.sh
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
M testdata/workloads/functional-planner/queries/PlannerTest/views.test
M testdata/workloads/functional-query/queries/QueryTest/compute-stats.test
M testdata/workloads/functional-query/queries/QueryTest/views.test
16 files changed, 350 insertions(+), 10 deletions(-)

Approvals:
  Impala Public Jenkins: Verified
  Quanlong Huang: Looks good to me, approved

-- 
To view, visit http://gerrit.cloudera.org:8080/17595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Gerrit-Change-Number: 17595
Gerrit-PatchSet: 16
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>

[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view

Posted by "Aman Sinha (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Qifan Chen, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/17595

to look at the new patch set (#13).

Change subject: IMPALA-10723: Treat materialized view as a table instead of a view
......................................................................

IMPALA-10723: Treat materialized view as a table instead of a view

The existing behavior is that materialized views are treated
as views and therefore expanded similar to a view when one
queries the MV directly (SELECT * FROM materialized_view).
This is incorrect since an MV is a regular table with physical
properties such as partitioning, clustering etc. and should be
treated as such even though it has a view definition associated
with it.

This patch focuses on the use case where MVs are created as
HDFS tables and makes the MVs a derived class of HdfsTable,
therefore making it a Table object. It adds support for
collecting and displaying statistics on materialized views
and these statistics could be leveraged by an external frontend
that supports MV based query rewrites (note that such a rewrite
is not supported by Impala with or without this patch). Note
that we are not introducing new syntax for MVs since DDL, DML
operations on MVs are only supported through Hive.

Directly querying a MV is permitted but inserts into MVs is not
since MVs are supposed to be only modified through an external
refresh when the source tables have modifications.

If the source tables associated with a materialized view have
column masking or row-filtering Ranger policies, querying the
MV will throw an error. This behavior is consistent with that
of Hive.

Testing:
 - Added transactional tables for alltypes, jointbl and used
   them as source tables to create materialized view.
 - Added tests for compute stats, drop stats, show stats and
   simple select query on a materialized view.
 - Added test for select on a materialized view when the
   source table has a column mask.
 - Modified analyzer tests related to alter, insert, drop of
   materialized view.

Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
---
M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java
A fe/src/main/java/org/apache/impala/catalog/MaterializedViewHdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java
M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java
M fe/src/test/java/org/apache/impala/authorization/AuthorizationTestBase.java
M fe/src/test/java/org/apache/impala/service/FrontendTest.java
M testdata/bin/compute-table-stats.sh
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
M testdata/workloads/functional-planner/queries/PlannerTest/views.test
M testdata/workloads/functional-query/queries/QueryTest/compute-stats.test
M testdata/workloads/functional-query/queries/QueryTest/views.test
17 files changed, 356 insertions(+), 10 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/95/17595/13
-- 
To view, visit http://gerrit.cloudera.org:8080/17595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Gerrit-Change-Number: 17595
Gerrit-PatchSet: 13
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>

[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view

Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/17595 )

Change subject: IMPALA-10723: Treat materialized view as a table instead of a view
......................................................................


Patch Set 7:

(3 comments)

Looks great!

http://gerrit.cloudera.org:8080/#/c/17595/7/fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
File fe/src/main/java/org/apache/impala/analysis/InsertStmt.java:

http://gerrit.cloudera.org:8080/#/c/17595/7/fe/src/main/java/org/apache/impala/analysis/InsertStmt.java@464
PS7, Line 464: table_ instanceof MaterializedViewHdfsTable
I wonder similar check should be done in ModifyStmt.java. (https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/analysis/ModifyStmt.java#L154)


http://gerrit.cloudera.org:8080/#/c/17595/7/fe/src/main/java/org/apache/impala/catalog/MaterializedViewHdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/MaterializedViewHdfsTable.java:

http://gerrit.cloudera.org:8080/#/c/17595/7/fe/src/main/java/org/apache/impala/catalog/MaterializedViewHdfsTable.java@70
PS7, Line 70: srcTablesAdded_
nit.  The test implies that there must be at least one table name added to srcTblNames_. The loop below goes over all src table names anyway. Maybe we can drop srcTablesAdded_?


http://gerrit.cloudera.org:8080/#/c/17595/7/testdata/workloads/functional-query/queries/QueryTest/compute-stats.test
File testdata/workloads/functional-query/queries/QueryTest/compute-stats.test:

http://gerrit.cloudera.org:8080/#/c/17595/7/testdata/workloads/functional-query/queries/QueryTest/compute-stats.test@1841
PS7, Line 1841: table
nit. May allow keyword MV here too.



-- 
To view, visit http://gerrit.cloudera.org:8080/17595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Gerrit-Change-Number: 17595
Gerrit-PatchSet: 7
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 24 Feb 2022 16:46:21 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10723: [WIP] Treat materialized view as a table instead of a view

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17595 )

Change subject: IMPALA-10723: [WIP] Treat materialized view as a table instead of a view
......................................................................


Patch Set 2:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/17595/2/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
File fe/src/main/java/org/apache/impala/analysis/Analyzer.java:

http://gerrit.cloudera.org:8080/#/c/17595/2/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@887
PS2, Line 887:         throw new AnalysisException(String.format("Querying the materialized view %s.%s " +
line too long (91 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/17595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Gerrit-Change-Number: 17595
Gerrit-PatchSet: 2
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 15 Jun 2021 07:57:54 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10723: [WIP] Treat materialized view as a table instead of a view

Posted by "Aman Sinha (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/17595

to look at the new patch set (#2).

Change subject: IMPALA-10723: [WIP] Treat materialized view as a table instead of a view
......................................................................

IMPALA-10723: [WIP] Treat materialized view as a table instead of a view

The existing behavior is that materialized views are treated
as views and therefore expanded similar to a view when one
queries the MV directly (SELECT * FROM materialized_view).
This is incorrect since an MV is a regular table with physical
properties such as partitioning, clustering etc. and should be
treated as such even though it has a view definition associated
with it.

This patch focuses on the use case where MVs are created as
HDFS tables and makes the MVs a derived class of HdfsTable,
therefore making it a Table object. It adds support for
collecting and displaying statistics on materialized views
and these statistics could be leveraged by an external frontend
that supports MV based query rewrites. Directly querying a MV
is permitted but inserts into MVs is not since MVs are supposed
to be only modified through an external refresh when the source
tables have modifications.

If the source tables associated with a materialized view have
column masking or row-filtering Ranger policies, querying the
MV will throw an error. This behavior is consistent with that
of Hive.

Testing
 - Added tests for compute stats, drop stats, show stats and
   simple select query on an MV.
 - Added a simple test for column masking of source table

TODO:
 - Change the exception thrown in the Analyzer

Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
---
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java
A fe/src/main/java/org/apache/impala/catalog/MaterializedViewHdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java
M testdata/workloads/functional-planner/queries/PlannerTest/views.test
M testdata/workloads/functional-query/queries/QueryTest/compute-stats.test
9 files changed, 205 insertions(+), 3 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/95/17595/2
-- 
To view, visit http://gerrit.cloudera.org:8080/17595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Gerrit-Change-Number: 17595
Gerrit-PatchSet: 2
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view

Posted by "Aman Sinha (Code Review)" <ge...@cloudera.org>.
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/17595 )

Change subject: IMPALA-10723: Treat materialized view as a table instead of a view
......................................................................


Patch Set 11:

(9 comments)

Thanks Quanlong and Qifan for the reviews. This patch had gone on my backburner for a while due to other priorities. I have addressed your review comments in the latest patchset.  Pls take another look.

http://gerrit.cloudera.org:8080/#/c/17595/5//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/17595/5//COMMIT_MSG@18
PS5, Line 18: makes the MVs a derived class of HdfsTable
> Hive can create Kudu external tables: https://cwiki.apache.org/confluence/d
Yes, for now this patch ignores Kudu. 
Regarding local catalog mode, I verified that the core functionality of querying and compute stats of the MV works when impala is started in the local_catalog mode.  The reason is that on the catalogd side, the MaterializedViewHdfsTable instance is created (derived class of HdfsTable) but on the impalad side,  LocalTable.loadTableMetadata() is called which receives a thrift response from the catalogd and creates the relevant LocalFsTable object. I have not checked the ranger authorization part in the local catalog mode but will do that in a follow-up jira if that's ok.


http://gerrit.cloudera.org:8080/#/c/17595/7/fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
File fe/src/main/java/org/apache/impala/analysis/InsertStmt.java:

http://gerrit.cloudera.org:8080/#/c/17595/7/fe/src/main/java/org/apache/impala/analysis/InsertStmt.java@464
PS7, Line 464: table_ instanceof MaterializedViewHdfsTable
> I wonder similar check should be done in ModifyStmt.java. (https://github.c
Impala only allows updates/deletes on Kudu tables, so if you do an UPDATE on an MV, it will actually fall through to the following line:
https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/analysis/ModifyStmt.java#L164
Which prints "Impala does not support modifying a non-Kudu table"
This is true for regular tables and MVs just inherit that behavior.


http://gerrit.cloudera.org:8080/#/c/17595/5/fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java
File fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java:

http://gerrit.cloudera.org:8080/#/c/17595/5/fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java@312
PS5, Line 312:       if (!tbl.isLoaded()) {
> Could you add a comment for this? It seems MaterializedViewHdfsTable could 
Yes, after further analysis, these 2 lines are not needed anymore, so I have removed them. I think there was an intermediate phase of the patch where I needed this since the first invocation of this function would encounter an IncompleteTable object but subsequently the concrete table object is created.


http://gerrit.cloudera.org:8080/#/c/17595/5/fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java@380
PS5, Line 380:     return tableNames;
> nit: blank line
Done


http://gerrit.cloudera.org:8080/#/c/17595/5/fe/src/main/java/org/apache/impala/catalog/MaterializedViewHdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/MaterializedViewHdfsTable.java:

http://gerrit.cloudera.org:8080/#/c/17595/5/fe/src/main/java/org/apache/impala/catalog/MaterializedViewHdfsTable.java@75
PS5, Line 75: TableMask tableMask = new Tab
> Should we report errors on this case?
I left it there as I wasn't sure if it could be a valid state . I will do some follow up to see if adding a preconditions check is ok.


http://gerrit.cloudera.org:8080/#/c/17595/5/fe/src/main/java/org/apache/impala/catalog/MaterializedViewHdfsTable.java@81
PS5, Line 81: 
> I think we should not return here if needsMaskingOrFiltering is false.
Good point.  Corrected it.


http://gerrit.cloudera.org:8080/#/c/17595/7/fe/src/main/java/org/apache/impala/catalog/MaterializedViewHdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/MaterializedViewHdfsTable.java:

http://gerrit.cloudera.org:8080/#/c/17595/7/fe/src/main/java/org/apache/impala/catalog/MaterializedViewHdfsTable.java@70
PS7, Line 70: (db);
> nit.  The test implies that there must be at least one table name added to 
I was being paranoid initially but you're right.. it can be removed.  Made this change.


http://gerrit.cloudera.org:8080/#/c/17595/5/fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java
File fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java:

http://gerrit.cloudera.org:8080/#/c/17595/5/fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java@3110
PS5, Line 3110:                 onServer(TPrivilegeLevel.ALL));
> Yes, I was in fact thinking of expanding the scope of the tests since previ
Added a new MV called mv1_alltypes_jointbl which has a join of 2 new source tables: alltypes_transactional and jointbl_transactional.  I needed to create a transactional version of these tables because MVs can only be defined on transactional source tables.
In this authorization test I have created a column masking policy on one of the source tables and querying the MV should fail.


http://gerrit.cloudera.org:8080/#/c/17595/7/testdata/workloads/functional-query/queries/QueryTest/compute-stats.test
File testdata/workloads/functional-query/queries/QueryTest/compute-stats.test:

http://gerrit.cloudera.org:8080/#/c/17595/7/testdata/workloads/functional-query/queries/QueryTest/compute-stats.test@1841
PS7, Line 1841: table
> nit. May allow keyword MV here too.
You mean 'show materialized view  stats .. ' ?  Since this is now treated as a table, I  think it is ok to not add a new keyword to the syntax here.



-- 
To view, visit http://gerrit.cloudera.org:8080/17595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Gerrit-Change-Number: 17595
Gerrit-PatchSet: 11
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 23 Mar 2022 03:24:16 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view

Posted by "Aman Sinha (Code Review)" <ge...@cloudera.org>.
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/17595 )

Change subject: IMPALA-10723: Treat materialized view as a table instead of a view
......................................................................


Patch Set 12:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/17595/11/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
File fe/src/main/java/org/apache/impala/analysis/Analyzer.java:

http://gerrit.cloudera.org:8080/#/c/17595/11/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@225
PS11, Line 225: mvAuthException
> nit. missing suffix _ for these two new fields?
added the suffix.


http://gerrit.cloudera.org:8080/#/c/17595/5/fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java
File fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java:

http://gerrit.cloudera.org:8080/#/c/17595/5/fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java@3110
PS5, Line 3110:                 onServer(TPrivilegeLevel.ALL));
> Added a new MV called mv1_alltypes_jointbl which has a join of 2 new source
Done


http://gerrit.cloudera.org:8080/#/c/17595/12/testdata/datasets/functional/functional_schema_template.sql
File testdata/datasets/functional/functional_schema_template.sql:

http://gerrit.cloudera.org:8080/#/c/17595/12/testdata/datasets/functional/functional_schema_template.sql@3446
PS12, Line 3446: -- Create a duplicate materialized view because we want to test
Reviewers,  I added this duplicate MV in the patch set 12 because of reasons mentioned here. I thought it was a cleaner way since create materialized view requires Hive and can only be done in this initial load phase.



-- 
To view, visit http://gerrit.cloudera.org:8080/17595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Gerrit-Change-Number: 17595
Gerrit-PatchSet: 12
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 24 Mar 2022 01:41:38 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17595 )

Change subject: IMPALA-10723: Treat materialized view as a table instead of a view
......................................................................


Patch Set 15: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/17595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Gerrit-Change-Number: 17595
Gerrit-PatchSet: 15
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 14 Apr 2022 10:26:50 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view

Posted by "Aman Sinha (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/17595

to look at the new patch set (#4).

Change subject: IMPALA-10723: Treat materialized view as a table instead of a view
......................................................................

IMPALA-10723: Treat materialized view as a table instead of a view

The existing behavior is that materialized views are treated
as views and therefore expanded similar to a view when one
queries the MV directly (SELECT * FROM materialized_view).
This is incorrect since an MV is a regular table with physical
properties such as partitioning, clustering etc. and should be
treated as such even though it has a view definition associated
with it.

This patch focuses on the use case where MVs are created as
HDFS tables and makes the MVs a derived class of HdfsTable,
therefore making it a Table object. It adds support for
collecting and displaying statistics on materialized views
and these statistics could be leveraged by an external frontend
that supports MV based query rewrites. Directly querying a MV
is permitted but inserts into MVs is not since MVs are supposed
to be only modified through an external refresh when the source
tables have modifications.

If the source tables associated with a materialized view have
column masking or row-filtering Ranger policies, querying the
MV will throw an error. This behavior is consistent with that
of Hive.

Testing:
 - Added tests for compute stats, drop stats, show stats and
   simple select query on an MV.
 - Added test for select on a materialized view when the
   source table has a column mask.
 - Modified analyzer tests related to alter, insert, drop of
   materialized view.

Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
---
M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java
A fe/src/main/java/org/apache/impala/catalog/MaterializedViewHdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java
M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java
M fe/src/test/java/org/apache/impala/authorization/AuthorizationTestBase.java
M testdata/workloads/functional-planner/queries/PlannerTest/views.test
M testdata/workloads/functional-query/queries/QueryTest/compute-stats.test
12 files changed, 230 insertions(+), 6 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/95/17595/4
-- 
To view, visit http://gerrit.cloudera.org:8080/17595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Gerrit-Change-Number: 17595
Gerrit-PatchSet: 4
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>

[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17595 )

Change subject: IMPALA-10723: Treat materialized view as a table instead of a view
......................................................................


Patch Set 3:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/17595/3/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
File fe/src/main/java/org/apache/impala/analysis/Analyzer.java:

http://gerrit.cloudera.org:8080/#/c/17595/3/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@892
PS3, Line 892: 	  .isReferencesMaskedTables(authChecker, getCatalog(), getUser())) {
tab used for whitespace


http://gerrit.cloudera.org:8080/#/c/17595/3/fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java
File fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java:

http://gerrit.cloudera.org:8080/#/c/17595/3/fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java@628
PS3, Line 628:        	"access type is: READONLY");
tab used for whitespace


http://gerrit.cloudera.org:8080/#/c/17595/3/fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java@637
PS3, Line 637:        	"access type is: READONLY");
tab used for whitespace



-- 
To view, visit http://gerrit.cloudera.org:8080/17595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Gerrit-Change-Number: 17595
Gerrit-PatchSet: 3
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 16 Jun 2021 00:25:57 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view

Posted by "Aman Sinha (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Qifan Chen, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/17595

to look at the new patch set (#9).

Change subject: IMPALA-10723: Treat materialized view as a table instead of a view
......................................................................

IMPALA-10723: Treat materialized view as a table instead of a view

The existing behavior is that materialized views are treated
as views and therefore expanded similar to a view when one
queries the MV directly (SELECT * FROM materialized_view).
This is incorrect since an MV is a regular table with physical
properties such as partitioning, clustering etc. and should be
treated as such even though it has a view definition associated
with it.

This patch focuses on the use case where MVs are created as
HDFS tables and makes the MVs a derived class of HdfsTable,
therefore making it a Table object. It adds support for
collecting and displaying statistics on materialized views
and these statistics could be leveraged by an external frontend
that supports MV based query rewrites (note that such a rewrite
is not supported by Impala with or without this patch).

Directly querying a MV is permitted but inserts into MVs is not
since MVs are supposed to be only modified through an external
refresh when the source tables have modifications.

If the source tables associated with a materialized view have
column masking or row-filtering Ranger policies, querying the
MV will throw an error. This behavior is consistent with that
of Hive.

Testing:
 - Added transactional tables for alltypes, jointbl and used
   them as source tables to create materialized view.
 - Added tests for compute stats, drop stats, show stats and
   simple select query on a materialized view.
 - Added test for select on a materialized view when the
   source table has a column mask.
 - Modified analyzer tests related to alter, insert, drop of
   materialized view.

Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
---
M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java
A fe/src/main/java/org/apache/impala/catalog/MaterializedViewHdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java
M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java
M fe/src/test/java/org/apache/impala/authorization/AuthorizationTestBase.java
M fe/src/test/java/org/apache/impala/service/FrontendTest.java
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
M testdata/workloads/functional-planner/queries/PlannerTest/views.test
M testdata/workloads/functional-query/queries/QueryTest/compute-stats.test
M testdata/workloads/functional-query/queries/QueryTest/views.test
16 files changed, 308 insertions(+), 9 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/95/17595/9
-- 
To view, visit http://gerrit.cloudera.org:8080/17595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Gerrit-Change-Number: 17595
Gerrit-PatchSet: 9
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>

[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17595 )

Change subject: IMPALA-10723: Treat materialized view as a table instead of a view
......................................................................


Patch Set 10:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/10325/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Gerrit-Change-Number: 17595
Gerrit-PatchSet: 10
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 23 Mar 2022 02:02:55 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view

Posted by "Aman Sinha (Code Review)" <ge...@cloudera.org>.
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/17595 )

Change subject: IMPALA-10723: Treat materialized view as a table instead of a view
......................................................................


Patch Set 11:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/17595/7/testdata/workloads/functional-query/queries/QueryTest/compute-stats.test
File testdata/workloads/functional-query/queries/QueryTest/compute-stats.test:

http://gerrit.cloudera.org:8080/#/c/17595/7/testdata/workloads/functional-query/queries/QueryTest/compute-stats.test@1841
PS7, Line 1841: table
> Yeah, I am considering this too, as what it means to treat MVs as tables. I
Adding the MV syntax is not in scope for this patch because the DDLs for MV create/drop/alter are actually done through Hive, not through Impala.  That's the reason why in the load script functional_schema_template.sql, it is under the 'CREATE_HIVE' section:
+---- CREATE_HIVE
+CREATE MATERIALIZED VIEW IF NOT EXISTS {db_name}{db_suffix}.{table_name}

In general, MVs are full-ACID tables (although insert-only is also allowed).  Impala does not support writing to full-ACID tables but it can read from it.  So, the main focus of this patch is to allow read operations on MVs including computing stats.  This allows an external frontend to leverage these stats while doing all the complex query rewrite operations and MV refresh (full/incremental based on the delta change in the source tables) via the external frontend.  

I can clarify this in the commit message in case that would make the scope clearer.



-- 
To view, visit http://gerrit.cloudera.org:8080/17595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Gerrit-Change-Number: 17595
Gerrit-PatchSet: 11
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 23 Mar 2022 23:08:51 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17595 )

Change subject: IMPALA-10723: Treat materialized view as a table instead of a view
......................................................................


Patch Set 15:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/10448/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Gerrit-Change-Number: 17595
Gerrit-PatchSet: 15
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 14 Apr 2022 05:23:34 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17595 )

Change subject: IMPALA-10723: Treat materialized view as a table instead of a view
......................................................................


Patch Set 14:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/10421/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Gerrit-Change-Number: 17595
Gerrit-PatchSet: 14
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Mon, 11 Apr 2022 04:47:39 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17595 )

Change subject: IMPALA-10723: Treat materialized view as a table instead of a view
......................................................................


Patch Set 5:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/8934/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Gerrit-Change-Number: 17595
Gerrit-PatchSet: 5
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 16 Jun 2021 01:49:16 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17595 )

Change subject: IMPALA-10723: Treat materialized view as a table instead of a view
......................................................................


Patch Set 3:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/8931/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Gerrit-Change-Number: 17595
Gerrit-PatchSet: 3
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 16 Jun 2021 00:46:01 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view

Posted by "Aman Sinha (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/17595

to look at the new patch set (#3).

Change subject: IMPALA-10723: Treat materialized view as a table instead of a view
......................................................................

IMPALA-10723: Treat materialized view as a table instead of a view

The existing behavior is that materialized views are treated
as views and therefore expanded similar to a view when one
queries the MV directly (SELECT * FROM materialized_view).
This is incorrect since an MV is a regular table with physical
properties such as partitioning, clustering etc. and should be
treated as such even though it has a view definition associated
with it.

This patch focuses on the use case where MVs are created as
HDFS tables and makes the MVs a derived class of HdfsTable,
therefore making it a Table object. It adds support for
collecting and displaying statistics on materialized views
and these statistics could be leveraged by an external frontend
that supports MV based query rewrites. Directly querying a MV
is permitted but inserts into MVs is not since MVs are supposed
to be only modified through an external refresh when the source
tables have modifications.

If the source tables associated with a materialized view have
column masking or row-filtering Ranger policies, querying the
MV will throw an error. This behavior is consistent with that
of Hive.

Testing:
 - Added tests for compute stats, drop stats, show stats and
   simple select query on an MV.
 - Added test for select on a materialized view when the
   source table has a column mask.
 - Modified analyzer tests related to alter, insert, drop of
   materialized view.

Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
---
M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java
A fe/src/main/java/org/apache/impala/catalog/MaterializedViewHdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java
M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java
M fe/src/test/java/org/apache/impala/authorization/AuthorizationTestBase.java
M testdata/workloads/functional-planner/queries/PlannerTest/views.test
M testdata/workloads/functional-query/queries/QueryTest/compute-stats.test
12 files changed, 233 insertions(+), 6 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/95/17595/3
-- 
To view, visit http://gerrit.cloudera.org:8080/17595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Gerrit-Change-Number: 17595
Gerrit-PatchSet: 3
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view

Posted by "Aman Sinha (Code Review)" <ge...@cloudera.org>.
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/17595 )

Change subject: IMPALA-10723: Treat materialized view as a table instead of a view
......................................................................


Patch Set 5:

(2 comments)

Thanks Quanlong for the review. I will do some investigation of the LocalCatalog mode and address your review comments in the follow up.

http://gerrit.cloudera.org:8080/#/c/17595/5//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/17595/5//COMMIT_MSG@18
PS5, Line 18: makes the MVs a derived class of HdfsTable
> 'HdfsTable' is not used in LocalCatalog mode coordinators. Does this patch 
It's good point about LocalCatalog mode. I will check it out and get back on this. 
Regarding MVs with Kudu, I doubt if that is a supported storage since the CREATE MATERIALIZED VIEW is only supported via Hive and Hive does not have Kudu as a storage option.


http://gerrit.cloudera.org:8080/#/c/17595/5/fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java
File fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java:

http://gerrit.cloudera.org:8080/#/c/17595/5/fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java@3110
PS5, Line 3110:         onTable("functional", "materialized_view", TPrivilegeLevel.SELECT));
> Could you add a more complex test that the number of srcTables of a MV is l
Yes, I was in fact thinking of expanding the scope of the tests since previously I just used the materialized_view table that was already present.



-- 
To view, visit http://gerrit.cloudera.org:8080/17595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Gerrit-Change-Number: 17595
Gerrit-PatchSet: 5
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Sat, 19 Jun 2021 01:10:38 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view

Posted by "Aman Sinha (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Qifan Chen, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/17595

to look at the new patch set (#15).

Change subject: IMPALA-10723: Treat materialized view as a table instead of a view
......................................................................

IMPALA-10723: Treat materialized view as a table instead of a view

The existing behavior is that materialized views are treated
as views and therefore expanded similar to a view when one
queries the MV directly (SELECT * FROM materialized_view).
This is incorrect since an MV is a regular table with physical
properties such as partitioning, clustering etc. and should be
treated as such even though it has a view definition associated
with it.

This patch focuses on the use case where MVs are created as
HDFS tables and makes the MVs a derived class of HdfsTable,
therefore making it a Table object. It adds support for
collecting and displaying statistics on materialized views
and these statistics could be leveraged by an external frontend
that supports MV based query rewrites (note that such a rewrite
is not supported by Impala with or without this patch). Note
that we are not introducing new syntax for MVs since DDL, DML
operations on MVs are only supported through Hive.

Directly querying a MV is permitted but inserts into MVs is not
since MVs are supposed to be only modified through an external
refresh when the source tables have modifications.

If the source tables associated with a materialized view have
column masking or row-filtering Ranger policies, querying the
MV will throw an error. This behavior is consistent with that
of Hive.

Testing:
 - Added transactional tables for alltypes, jointbl and used
   them as source tables to create materialized view.
 - Added tests for compute stats, drop stats, show stats and
   simple select query on a materialized view.
 - Added test for select on a materialized view when the
   source table has a column mask.
 - Modified analyzer tests related to alter, insert, drop of
   materialized view.

Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
---
M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java
A fe/src/main/java/org/apache/impala/catalog/MaterializedViewHdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java
M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java
M fe/src/test/java/org/apache/impala/authorization/AuthorizationTestBase.java
M testdata/bin/compute-table-stats.sh
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
M testdata/workloads/functional-planner/queries/PlannerTest/views.test
M testdata/workloads/functional-query/queries/QueryTest/compute-stats.test
M testdata/workloads/functional-query/queries/QueryTest/views.test
16 files changed, 350 insertions(+), 10 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/95/17595/15
-- 
To view, visit http://gerrit.cloudera.org:8080/17595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Gerrit-Change-Number: 17595
Gerrit-PatchSet: 15
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>

[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17595 )

Change subject: IMPALA-10723: Treat materialized view as a table instead of a view
......................................................................


Patch Set 8: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7948/


-- 
To view, visit http://gerrit.cloudera.org:8080/17595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Gerrit-Change-Number: 17595
Gerrit-PatchSet: 8
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Mon, 21 Mar 2022 03:12:32 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view

Posted by "Aman Sinha (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Qifan Chen, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/17595

to look at the new patch set (#14).

Change subject: IMPALA-10723: Treat materialized view as a table instead of a view
......................................................................

IMPALA-10723: Treat materialized view as a table instead of a view

The existing behavior is that materialized views are treated
as views and therefore expanded similar to a view when one
queries the MV directly (SELECT * FROM materialized_view).
This is incorrect since an MV is a regular table with physical
properties such as partitioning, clustering etc. and should be
treated as such even though it has a view definition associated
with it.

This patch focuses on the use case where MVs are created as
HDFS tables and makes the MVs a derived class of HdfsTable,
therefore making it a Table object. It adds support for
collecting and displaying statistics on materialized views
and these statistics could be leveraged by an external frontend
that supports MV based query rewrites (note that such a rewrite
is not supported by Impala with or without this patch). Note
that we are not introducing new syntax for MVs since DDL, DML
operations on MVs are only supported through Hive.

Directly querying a MV is permitted but inserts into MVs is not
since MVs are supposed to be only modified through an external
refresh when the source tables have modifications.

If the source tables associated with a materialized view have
column masking or row-filtering Ranger policies, querying the
MV will throw an error. This behavior is consistent with that
of Hive.

Testing:
 - Added transactional tables for alltypes, jointbl and used
   them as source tables to create materialized view.
 - Added tests for compute stats, drop stats, show stats and
   simple select query on a materialized view.
 - Added test for select on a materialized view when the
   source table has a column mask.
 - Modified analyzer tests related to alter, insert, drop of
   materialized view.

Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
---
M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java
A fe/src/main/java/org/apache/impala/catalog/MaterializedViewHdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java
M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java
M fe/src/test/java/org/apache/impala/authorization/AuthorizationTestBase.java
M testdata/bin/compute-table-stats.sh
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
M testdata/workloads/functional-planner/queries/PlannerTest/views.test
M testdata/workloads/functional-query/queries/QueryTest/compute-stats.test
M testdata/workloads/functional-query/queries/QueryTest/views.test
16 files changed, 345 insertions(+), 9 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/95/17595/14
-- 
To view, visit http://gerrit.cloudera.org:8080/17595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Gerrit-Change-Number: 17595
Gerrit-PatchSet: 14
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>

[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17595 )

Change subject: IMPALA-10723: Treat materialized view as a table instead of a view
......................................................................


Patch Set 14:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8033/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/17595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Gerrit-Change-Number: 17595
Gerrit-PatchSet: 14
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Mon, 11 Apr 2022 05:04:04 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10723: [WIP] Treat materialized view as a table instead of a view

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17595 )

Change subject: IMPALA-10723: [WIP] Treat materialized view as a table instead of a view
......................................................................


Patch Set 1:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/17595/1/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
File fe/src/main/java/org/apache/impala/analysis/Analyzer.java:

http://gerrit.cloudera.org:8080/#/c/17595/1/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@887
PS1, Line 887:         throw new AnalysisException(String.format("Querying the materialized view %s.%s " +
line too long (91 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/17595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Gerrit-Change-Number: 17595
Gerrit-PatchSet: 1
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 15 Jun 2021 07:03:48 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/17595 )

Change subject: IMPALA-10723: Treat materialized view as a table instead of a view
......................................................................


Patch Set 5:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/17595/5//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/17595/5//COMMIT_MSG@18
PS5, Line 18: makes the MVs a derived class of HdfsTable
> It's good point about LocalCatalog mode. I will check it out and get back o
Hive can create Kudu external tables: https://cwiki.apache.org/confluence/display/Hive/Kudu+Integration
However, when trying to create a MV on kudu, I got an error:

 Error: Error while compiling statement: FAILED: SemanticException org.apache.hadoop.hive.ql.parse.SemanticException: Automatic rewriting for materialized view cannot be enabled if the materialized view uses non-transactional tables (state=42000,code=40000)

So I think currently it's ok to ignore Kudu. Note that Hive supports MV on Druid: https://cwiki.apache.org/confluence/display/Hive/Materialized+views
I think we should prepare for the extension when one day Hive supports MV on Kudu.



-- 
To view, visit http://gerrit.cloudera.org:8080/17595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Gerrit-Change-Number: 17595
Gerrit-PatchSet: 5
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Mon, 21 Jun 2021 03:18:11 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view

Posted by "Aman Sinha (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Qifan Chen, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/17595

to look at the new patch set (#8).

Change subject: IMPALA-10723: Treat materialized view as a table instead of a view
......................................................................

IMPALA-10723: Treat materialized view as a table instead of a view

The existing behavior is that materialized views are treated
as views and therefore expanded similar to a view when one
queries the MV directly (SELECT * FROM materialized_view).
This is incorrect since an MV is a regular table with physical
properties such as partitioning, clustering etc. and should be
treated as such even though it has a view definition associated
with it.

This patch focuses on the use case where MVs are created as
HDFS tables and makes the MVs a derived class of HdfsTable,
therefore making it a Table object. It adds support for
collecting and displaying statistics on materialized views
and these statistics could be leveraged by an external frontend
that supports MV based query rewrites (note that such a rewrite
is not supported by Impala with or without this patch).

Directly querying a MV is permitted but inserts into MVs is not
since MVs are supposed to be only modified through an external
refresh when the source tables have modifications.

If the source tables associated with a materialized view have
column masking or row-filtering Ranger policies, querying the
MV will throw an error. This behavior is consistent with that
of Hive.

Testing:
 - Added transactional tables for alltypes, jointbl and used
   them as source tables to create materialized view.
 - Added tests for compute stats, drop stats, show stats and
   simple select query on a materialized view.
 - Added test for select on a materialized view when the
   source table has a column mask.
 - Modified analyzer tests related to alter, insert, drop of
   materialized view.

Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
---
M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java
A fe/src/main/java/org/apache/impala/catalog/MaterializedViewHdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java
M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java
M fe/src/test/java/org/apache/impala/authorization/AuthorizationTestBase.java
M fe/src/test/java/org/apache/impala/service/FrontendTest.java
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
M testdata/workloads/functional-planner/queries/PlannerTest/views.test
M testdata/workloads/functional-query/queries/QueryTest/compute-stats.test
M testdata/workloads/functional-query/queries/QueryTest/views.test
16 files changed, 319 insertions(+), 10 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/95/17595/8
-- 
To view, visit http://gerrit.cloudera.org:8080/17595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Gerrit-Change-Number: 17595
Gerrit-PatchSet: 8
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>

[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17595 )

Change subject: IMPALA-10723: Treat materialized view as a table instead of a view
......................................................................


Patch Set 9:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/10313/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Gerrit-Change-Number: 17595
Gerrit-PatchSet: 9
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Mon, 21 Mar 2022 06:53:48 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view

Posted by "Aman Sinha (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Qifan Chen, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/17595

to look at the new patch set (#11).

Change subject: IMPALA-10723: Treat materialized view as a table instead of a view
......................................................................

IMPALA-10723: Treat materialized view as a table instead of a view

The existing behavior is that materialized views are treated
as views and therefore expanded similar to a view when one
queries the MV directly (SELECT * FROM materialized_view).
This is incorrect since an MV is a regular table with physical
properties such as partitioning, clustering etc. and should be
treated as such even though it has a view definition associated
with it.

This patch focuses on the use case where MVs are created as
HDFS tables and makes the MVs a derived class of HdfsTable,
therefore making it a Table object. It adds support for
collecting and displaying statistics on materialized views
and these statistics could be leveraged by an external frontend
that supports MV based query rewrites (note that such a rewrite
is not supported by Impala with or without this patch).

Directly querying a MV is permitted but inserts into MVs is not
since MVs are supposed to be only modified through an external
refresh when the source tables have modifications.

If the source tables associated with a materialized view have
column masking or row-filtering Ranger policies, querying the
MV will throw an error. This behavior is consistent with that
of Hive.

Testing:
 - Added transactional tables for alltypes, jointbl and used
   them as source tables to create materialized view.
 - Added tests for compute stats, drop stats, show stats and
   simple select query on a materialized view.
 - Added test for select on a materialized view when the
   source table has a column mask.
 - Modified analyzer tests related to alter, insert, drop of
   materialized view.

Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
---
M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java
A fe/src/main/java/org/apache/impala/catalog/MaterializedViewHdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java
M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java
M fe/src/test/java/org/apache/impala/authorization/AuthorizationTestBase.java
M fe/src/test/java/org/apache/impala/service/FrontendTest.java
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
M testdata/workloads/functional-planner/queries/PlannerTest/views.test
M testdata/workloads/functional-query/queries/QueryTest/compute-stats.test
M testdata/workloads/functional-query/queries/QueryTest/views.test
16 files changed, 313 insertions(+), 9 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/95/17595/11
-- 
To view, visit http://gerrit.cloudera.org:8080/17595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Gerrit-Change-Number: 17595
Gerrit-PatchSet: 11
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>

[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view

Posted by "Aman Sinha (Code Review)" <ge...@cloudera.org>.
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/17595 )

Change subject: IMPALA-10723: Treat materialized view as a table instead of a view
......................................................................


Patch Set 12:

> Patch Set 12: Verified-1
> 
> Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7961/

A few of the new tests I added for this patch failed due to the materialized view table being empty. On my local machine it works fine.  I looked at the logs from the jenkins run and see that the CREATE MATERIALIZED VIEW .. FROM alltypes_transactional join jointbl_transactional ... is being executed even before the INSERT commands into those source tables are executed, so the MV remains empty.  I will check if this can be addressed through a DEPENDENT_LOAD_HIVE section.  Not sure if this is applicable for materialized views.


-- 
To view, visit http://gerrit.cloudera.org:8080/17595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Gerrit-Change-Number: 17595
Gerrit-PatchSet: 12
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 24 Mar 2022 07:01:18 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view

Posted by "Aman Sinha (Code Review)" <ge...@cloudera.org>.
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/17595 )

Change subject: IMPALA-10723: Treat materialized view as a table instead of a view
......................................................................


Patch Set 13:

> Patch Set 11:
> 
> (1 comment)

Added the DEPENDENT_HIVE_LOAD section in the load script for the materialized views. Will test by doing a dry run on jenkins.


-- 
To view, visit http://gerrit.cloudera.org:8080/17595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Gerrit-Change-Number: 17595
Gerrit-PatchSet: 13
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 06 Apr 2022 06:43:40 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17595 )

Change subject: IMPALA-10723: Treat materialized view as a table instead of a view
......................................................................


Patch Set 15:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8046/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/17595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Gerrit-Change-Number: 17595
Gerrit-PatchSet: 15
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 14 Apr 2022 06:02:33 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17595 )

Change subject: IMPALA-10723: Treat materialized view as a table instead of a view
......................................................................


Patch Set 4:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/17595/4/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
File fe/src/main/java/org/apache/impala/analysis/Analyzer.java:

http://gerrit.cloudera.org:8080/#/c/17595/4/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@892
PS4, Line 892: 	  .isReferencesMaskedTables(authChecker, getCatalog(), getUser())) {
tab used for whitespace



-- 
To view, visit http://gerrit.cloudera.org:8080/17595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Gerrit-Change-Number: 17595
Gerrit-PatchSet: 4
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 16 Jun 2021 01:28:12 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10723: [WIP] Treat materialized view as a table instead of a view

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17595 )

Change subject: IMPALA-10723: [WIP] Treat materialized view as a table instead of a view
......................................................................


Patch Set 1:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/8915/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Gerrit-Change-Number: 17595
Gerrit-PatchSet: 1
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 15 Jun 2021 07:28:30 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/17595 )

Change subject: IMPALA-10723: Treat materialized view as a table instead of a view
......................................................................


Patch Set 5:

(6 comments)

http://gerrit.cloudera.org:8080/#/c/17595/5//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/17595/5//COMMIT_MSG@18
PS5, Line 18: makes the MVs a derived class of HdfsTable
'HdfsTable' is not used in LocalCatalog mode coordinators. Does this patch work in LocalCatalog mode?

BTW, does MV support using Kudu as the storage?


http://gerrit.cloudera.org:8080/#/c/17595/5/fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java
File fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java:

http://gerrit.cloudera.org:8080/#/c/17595/5/fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java@312
PS5, Line 312:         if (!(tbl instanceof MaterializedViewHdfsTable)) {
Could you add a comment for this? It seems MaterializedViewHdfsTable could not go here since it extends HdfsTable which extends CatalogObjectImpl whose isLoaded() always return true.


http://gerrit.cloudera.org:8080/#/c/17595/5/fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java@380
PS5, Line 380: 
nit: blank line


http://gerrit.cloudera.org:8080/#/c/17595/5/fe/src/main/java/org/apache/impala/catalog/MaterializedViewHdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/MaterializedViewHdfsTable.java:

http://gerrit.cloudera.org:8080/#/c/17595/5/fe/src/main/java/org/apache/impala/catalog/MaterializedViewHdfsTable.java@75
PS5, Line 75: if (srcTbl == null) continue;
Should we report errors on this case?


http://gerrit.cloudera.org:8080/#/c/17595/5/fe/src/main/java/org/apache/impala/catalog/MaterializedViewHdfsTable.java@81
PS5, Line 81: return needsMaskingOrFiltering;
I think we should not return here if needsMaskingOrFiltering is false.


http://gerrit.cloudera.org:8080/#/c/17595/5/fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java
File fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java:

http://gerrit.cloudera.org:8080/#/c/17595/5/fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java@3110
PS5, Line 3110:         onTable("functional", "materialized_view", TPrivilegeLevel.SELECT));
Could you add a more complex test that the number of srcTables of a MV is larger than one?



-- 
To view, visit http://gerrit.cloudera.org:8080/17595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Gerrit-Change-Number: 17595
Gerrit-PatchSet: 5
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 18 Jun 2021 05:53:38 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17595 )

Change subject: IMPALA-10723: Treat materialized view as a table instead of a view
......................................................................


Patch Set 4:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/8933/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Gerrit-Change-Number: 17595
Gerrit-PatchSet: 4
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 16 Jun 2021 01:51:29 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17595 )

Change subject: IMPALA-10723: Treat materialized view as a table instead of a view
......................................................................


Patch Set 11:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/10326/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Gerrit-Change-Number: 17595
Gerrit-PatchSet: 11
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 23 Mar 2022 02:49:28 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view

Posted by "Aman Sinha (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Qifan Chen, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/17595

to look at the new patch set (#12).

Change subject: IMPALA-10723: Treat materialized view as a table instead of a view
......................................................................

IMPALA-10723: Treat materialized view as a table instead of a view

The existing behavior is that materialized views are treated
as views and therefore expanded similar to a view when one
queries the MV directly (SELECT * FROM materialized_view).
This is incorrect since an MV is a regular table with physical
properties such as partitioning, clustering etc. and should be
treated as such even though it has a view definition associated
with it.

This patch focuses on the use case where MVs are created as
HDFS tables and makes the MVs a derived class of HdfsTable,
therefore making it a Table object. It adds support for
collecting and displaying statistics on materialized views
and these statistics could be leveraged by an external frontend
that supports MV based query rewrites (note that such a rewrite
is not supported by Impala with or without this patch). Note
that we are not introducing new syntax for MVs since DDL, DML
operations on MVs are only supported through Hive.

Directly querying a MV is permitted but inserts into MVs is not
since MVs are supposed to be only modified through an external
refresh when the source tables have modifications.

If the source tables associated with a materialized view have
column masking or row-filtering Ranger policies, querying the
MV will throw an error. This behavior is consistent with that
of Hive.

Testing:
 - Added transactional tables for alltypes, jointbl and used
   them as source tables to create materialized view.
 - Added tests for compute stats, drop stats, show stats and
   simple select query on a materialized view.
 - Added test for select on a materialized view when the
   source table has a column mask.
 - Modified analyzer tests related to alter, insert, drop of
   materialized view.

Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
---
M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java
A fe/src/main/java/org/apache/impala/catalog/MaterializedViewHdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java
M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java
M fe/src/test/java/org/apache/impala/authorization/AuthorizationTestBase.java
M fe/src/test/java/org/apache/impala/service/FrontendTest.java
M testdata/bin/compute-table-stats.sh
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
M testdata/workloads/functional-planner/queries/PlannerTest/views.test
M testdata/workloads/functional-query/queries/QueryTest/compute-stats.test
M testdata/workloads/functional-query/queries/QueryTest/views.test
17 files changed, 346 insertions(+), 10 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/95/17595/12
-- 
To view, visit http://gerrit.cloudera.org:8080/17595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Gerrit-Change-Number: 17595
Gerrit-PatchSet: 12
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>

[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view

Posted by "Aman Sinha (Code Review)" <ge...@cloudera.org>.
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/17595 )

Change subject: IMPALA-10723: Treat materialized view as a table instead of a view
......................................................................


Patch Set 14:

> Patch Set 13:
> 
> (1 comment)
> 
> > Patch Set 13: Verified-1
> > 
> > Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8026/
> 
> I take a look at file-list-begin-1.log in one of the build. There are no files under the functional_parquet.alltypes_transactional and functional_parquet.jointbl_transactional tables. Neither can I find statements loading these two tables in logs/data_loading/sql/functional/load-functional-query-exhaustive-impala-generated-parquet-none-none.sql.
> 
> Note that these tables under the functional db is ok (non-empty). However, I can't find the rebuild statement on the MV in logs/data_loading/sql/functional/load-functional-query-exhaustive-hive-generated-text-none-none.sql.

In the latest patch set (PS14) I made a few changes:
 1. For the new transactional tables and materialized views, I am now doing 'STORED AS {file_format}'. Previously this was doing STORED AS PARQUET.
 2. In datasets/functional/schema_constraints.csv, I am restricting the MVs
 and the source tables to the ORC format. The reason is that ALTER MATERIALIZED VIEW ... REBUILD is not rebuilding the MV if the format is Parquet but does it for ORC. The log file shows the SQL command to do the rebuild is issued but the Tez tasks never get spawned and no error is produced. On the other hand for ORC, I do see the Tez tasks. The MV rebuild is supposed to work for both ORC and Parquet but perhaps it is not supported in the Hive version in the toolchain. I can look into this as a follow-up. 
 3. There was an interesting side-effect of the 2 MVs. Since they both had the exact same SQL definition, what would happen is that when creating the second MV mv2_alltypes_jointbl, if the first MV mv1_alltypes_jointbl had already been created, the optimizer would do a rewrite. This is not the intent, so I have modified the mv2_alltypes_jointbl definition slightly.


-- 
To view, visit http://gerrit.cloudera.org:8080/17595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Gerrit-Change-Number: 17595
Gerrit-PatchSet: 14
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Mon, 11 Apr 2022 04:47:36 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view

Posted by "Aman Sinha (Code Review)" <ge...@cloudera.org>.
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/17595 )

Change subject: IMPALA-10723: Treat materialized view as a table instead of a view
......................................................................


Patch Set 15:

> Patch Set 15:
> 
> > Patch Set 14:
> > 
> > (1 comment)
> 
> Coincidentally, that's what I concluded just a little while ago after playing around with the compute stats.  On my local machine I was getting 503B consistently for mv2_alltypes_jointbl.  I have replaced with the regex.

Since this was the only failure on the last run, I expect the tests to pass.


-- 
To view, visit http://gerrit.cloudera.org:8080/17595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Gerrit-Change-Number: 17595
Gerrit-PatchSet: 15
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 14 Apr 2022 05:11:58 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/17595 )

Change subject: IMPALA-10723: Treat materialized view as a table instead of a view
......................................................................


Patch Set 14:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/17595/14/testdata/workloads/functional-query/queries/QueryTest/compute-stats.test
File testdata/workloads/functional-query/queries/QueryTest/compute-stats.test:

http://gerrit.cloudera.org:8080/#/c/17595/14/testdata/workloads/functional-query/queries/QueryTest/compute-stats.test@1849
PS14, Line 1849: '503B'
We'd better replace this with regex:.+B (Line 532 is an example).
The file size might not be consistent across different builds. The failures actually show this
https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/16250
https://jenkins.impala.io/job/ubuntu-16.04-dockerised-tests/5550



-- 
To view, visit http://gerrit.cloudera.org:8080/17595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Gerrit-Change-Number: 17595
Gerrit-PatchSet: 14
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 14 Apr 2022 04:58:25 +0000
Gerrit-HasComments: Yes