You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "anshuksi282-ksolves (via GitHub)" <gi...@apache.org> on 2026/07/28 10:08:57 UTC
[PR] HIVE-27636. Exception in HiveMaterializedViewsRegistry is leaving staging directories behind [hive]
anshuksi282-ksolves opened a new pull request, #6650:
URL: https://github.com/apache/hive/pull/6650
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/Hive/HowToContribute
2. Ensure that you have created an issue on the Hive project JIRA: https://issues.apache.org/jira/projects/HIVE/summary
3. Ensure you have added or run the appropriate tests for your PR:
4. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]HIVE-XXXXX: Your PR title ...'.
5. Be sure to keep the PR description updated to reflect all changes.
6. Please write your PR title to summarize what this PR proposes.
7. If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
`HiveMaterializedViewsRegistry.createMaterialization()` creates a `Context` object (used for query parsing/semantic analysis) before attempting to parse the materialized view's query. If parsing or semantic analysis fails, the method logs a warning and returns `null`, but never called `context.clear()` to clean up any scratch/staging directories the `Context` may have created during that attempt.
This PR adds a call to `context.clear()` inside the `catch` block, so that any such directories are removed eagerly, at the point the exception is caught, rather than being left behind.
### Why are the changes needed?
Directories created by `Context` (e.g. for CTE materialization, or dummy-table handling for FROM-less queries) are only registered via `fs.deleteOnExit()`. This means they are only cleaned up when the JVM exits. For a long-running process like HiveServer2, if `createMaterialization()` repeatedly hits parse/semantic exceptions (e.g. due to a materialized view definition referencing a table that no longer exists), these directories accumulate over time and are never removed until the server restarts. This can eventually cause HDFS directory-item-limit errors, as reported in the JIRA:
Cannot create staging directory '...': The directory item limit of ... is exceeded: limit=1048576 items=1048576
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
- Added `TestHiveMaterializedViewsRegistry#testStagingDirIsCleanedUpOnParseException`. It uses a materialized view query with a CTE referenced enough times to trigger CTE materialization (which creates a real directory tracked by `Context`), followed by an invalid column reference that fails during semantic analysis. The test asserts that no per-query scratch subdirectory remains once `createMaterialization()` returns.
- Verified this test fails, with a leftover directory reported, when the fix is reverted, and passes with the fix applied.
- Added `TestContext#testClearRemovesScratchDirEagerlyWhenHDFSCleanupEnabled`, a more targeted test confirming that `Context.clear()` removes a scratch directory immediately (rather than relying on `deleteOnExit`) when `isHDFSCleanup` is enabled.
- `mvn -pl ql test -Dtest=TestContext,TestHiveMaterializedViewsRegistry` passes.
Contains content generated by Claude (Anthropic), used to help investigate the root cause, draft the fix, and design/verify the accompanying tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27636. Exception in HiveMaterializedViewsRegistry is leaving staging directories behind [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6650:
URL: https://github.com/apache/hive/pull/6650#issuecomment-5103727035
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6650) **Quality Gate passed**
Issues
 [8 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6650&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6650&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6650&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6650&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6650&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6650)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27636. Exception in HiveMaterializedViewsRegistry is leaving staging directories behind [hive]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6650:
URL: https://github.com/apache/hive/pull/6650#issuecomment-5861425049
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
Feel free to reach out on the dev@hive.apache.org list if the patch is in need of reviews.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27636. Exception in HiveMaterializedViewsRegistry is leaving staging directories behind [hive]
Posted by "rtrivedi12 (via GitHub)" <gi...@apache.org>.
rtrivedi12 commented on code in PR #6650:
URL: https://github.com/apache/hive/pull/6650#discussion_r3668254261
##########
ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMaterializedViewsRegistry.java:
##########
@@ -225,12 +242,19 @@ public HiveRelOptMaterialization createMaterialization(HiveConf conf, Table mate
" ignored; error creating view replacement");
return null;
}
+ final Context context = createContext(conf);
final CBOPlan plan;
try {
- plan = ParseUtils.parseQuery(createContext(conf), viewQuery);
+ plan = ParseUtils.parseQuery(context, viewQuery);
} catch (Exception e) {
LOG.warn("Materialized view " + materializedViewTable.getCompleteName() +
" ignored; error parsing original query; " + e);
+ try {
+ context.clear();
+ } catch (IOException ioe) {
+ LOG.warn("Error while cleaning up staging directories for materialized view " +
+ materializedViewTable.getCompleteName(), ioe);
+ }
return null;
Review Comment:
@anshuksi282-ksolves context.clear() is only called on exception, but I think the success path also abandons the Context without cleanup. Do you think we should have context.clear() in finally block so it is always called?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27636. Exception in HiveMaterializedViewsRegistry is leaving staging directories behind [hive]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #6650: HIVE-27636. Exception in HiveMaterializedViewsRegistry is leaving staging directories behind
URL: https://github.com/apache/hive/pull/6650
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27636. Exception in HiveMaterializedViewsRegistry is leaving staging directories behind [hive]
Posted by "anshuksi282-ksolves (via GitHub)" <gi...@apache.org>.
anshuksi282-ksolves commented on code in PR #6650:
URL: https://github.com/apache/hive/pull/6650#discussion_r3671891533
##########
ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMaterializedViewsRegistry.java:
##########
@@ -225,12 +242,19 @@ public HiveRelOptMaterialization createMaterialization(HiveConf conf, Table mate
" ignored; error creating view replacement");
return null;
}
+ final Context context = createContext(conf);
final CBOPlan plan;
try {
- plan = ParseUtils.parseQuery(createContext(conf), viewQuery);
+ plan = ParseUtils.parseQuery(context, viewQuery);
} catch (Exception e) {
LOG.warn("Materialized view " + materializedViewTable.getCompleteName() +
" ignored; error parsing original query; " + e);
+ try {
+ context.clear();
+ } catch (IOException ioe) {
+ LOG.warn("Error while cleaning up staging directories for materialized view " +
+ materializedViewTable.getCompleteName(), ioe);
+ }
return null;
Review Comment:
> @anshuksi282-ksolves context.clear() is only called on exception, but I think the success path also abandons the Context without cleanup. Do you think we should have context.clear() in finally block so it is always called?
Thanks for catching that, @rtrivedi12! You're right — moved context.clear() into a finally block now, so it's called on both the success and exception paths. Also fixed the checkstyle indentation issues flagged by SonarQube. Please take a look when you get a chance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27636. Exception in HiveMaterializedViewsRegistry is leaving staging directories behind [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6650:
URL: https://github.com/apache/hive/pull/6650#issuecomment-5115012446
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6650) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6650&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6650&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6650&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6650&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6650&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6650)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org