You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by GitBox <gi...@apache.org> on 2022/01/25 14:11:41 UTC
[GitHub] [hive] kasakrisz opened a new pull request #2972: Hive 24820 master mv cache fix2
kasakrisz opened a new pull request #2972:
URL: https://github.com/apache/hive/pull/2972
### What changes were proposed in this pull request?
1. Replace ConcurrentMap.compute to computeIfAbsent in MaterializedViewsCache.putIfAbsent
2. Add a new version of `HiveMaterializedViewsRegistry.refresh()`
3. When rebuilding Instead of adding a new instance of the same MV refresh the existing one in the cache.
### Why are the changes needed?
Using ConcurrentMap.compute enabled adding multiple entries of the same Materialization instance to the sqlText -> materialization map.
`HiveMaterializedViewsRegistry.refresh(HiveConf conf, Table oldMaterializedViewTable, Table materializedViewTable)` did not removed outdated cache entry since a newer version of the passed table object was loaded from MetaStore in `MaterializedViewUpdateOperation`.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
```
mvn test -Dtest=TestMaterializedViewsCache -pl ql
mvn test -Dtest.output.overwrite -DskipSparkTests -Dtest=TestMiniLlapLocalCliDriver -Dqfile=materialized_view_create_rewrite_3.q -pl itests/qtest -Pitests
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] kasakrisz merged pull request #2972: HIVE-24820: MaterializedViewCache enables adding multiple entries of the same Materialization instance
Posted by GitBox <gi...@apache.org>.
kasakrisz merged pull request #2972:
URL: https://github.com/apache/hive/pull/2972
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] kgyrtkirk commented on a change in pull request #2972: HIVE-24820: MaterializedViewCache enables adding multiple entries of the same Materialization instance
Posted by GitBox <gi...@apache.org>.
kgyrtkirk commented on a change in pull request #2972:
URL: https://github.com/apache/hive/pull/2972#discussion_r793433268
##########
File path: ql/src/test/queries/clientpositive/materialized_view_partitioned.q
##########
@@ -60,3 +60,7 @@ ALTER MATERIALIZED VIEW partition_mv_3 REBUILD;
ALTER MATERIALIZED VIEW partition_mv_3 REBUILD;
SELECT * FROM partition_mv_3 where key = 238;
+
+DROP MATERIALIZED VIEW partition_mv_1;
+DROP MATERIALIZED VIEW partition_mv_2;
+DROP MATERIALIZED VIEW partition_mv_3;
Review comment:
at some point you could probably add some cleanup to qtestutil for these around:
https://github.com/apache/hive/blob/69aec11e711293eaf5eb8a3fbb44905bc11841c9/itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java#L365
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] kasakrisz commented on a change in pull request #2972: HIVE-24820: MaterializedViewCache enables adding multiple entries of the same Materialization instance
Posted by GitBox <gi...@apache.org>.
kasakrisz commented on a change in pull request #2972:
URL: https://github.com/apache/hive/pull/2972#discussion_r793470869
##########
File path: ql/src/test/queries/clientpositive/materialized_view_partitioned.q
##########
@@ -60,3 +60,7 @@ ALTER MATERIALIZED VIEW partition_mv_3 REBUILD;
ALTER MATERIALIZED VIEW partition_mv_3 REBUILD;
SELECT * FROM partition_mv_3 where key = 238;
+
+DROP MATERIALIZED VIEW partition_mv_1;
+DROP MATERIALIZED VIEW partition_mv_2;
+DROP MATERIALIZED VIEW partition_mv_3;
Review comment:
Created a follow-up patch: https://issues.apache.org/jira/browse/HIVE-25906
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org