You are viewing a plain text version of this content. The canonical link for it is here.

Posted to gitbox@hive.apache.org by "scarlin-cloudera (via GitHub)" <gi...@apache.org> on 2024/04/03 19:59:35 UTC

[PR] HIVE-28173: Fixed staging dir issues on materialized views on HDFS en… [hive]

scarlin-cloudera opened a new pull request, #5180:
URL: https://github.com/apache/hive/pull/5180

…crypted tables

There are 2 issues involving staging directories on hdfs encrypted tables

1) The staging directory is created at compile time. For non hdfs encrypted tables, the "mkdir" flag is set to false.
There is no such flag for hdfs encrypted tables.

2) The "FileSystem.deleteOnFileExit()" method is not called from the HiveMaterializedViewRegistry thread.

For the first issue, the fix has only been implemented for the materialized view registry thread. It is a bit more risky to implement this for all compiled queries in the SemanticAnalyzer, but perhaps it is correct. However, the implemented fix in this commit is similar to previous implemented logic and a customer has this specific problem on their production machines so this fix is important.

The second issue is also specifically fixed for the materialized view registry thread. A small refactoring was done since the "isLoadingMaterializedView" flag should not be set within ParseUtils

<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/Hive/HowToContribute
2. Ensure that you have created an issue on the Hive project JIRA: https://issues.apache.org/jira/projects/HIVE/summary
3. Ensure you have added or run the appropriate tests for your PR:
4. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]HIVE-XXXXX: Your PR title ...'.
5. Be sure to keep the PR description updated to reflect all changes.
6. Please write your PR title to summarize what this PR proposes.
7. If possible, provide a concise example to reproduce the issue for a faster review.

-->

### What changes were proposed in this pull request?

Mentioned above

### Why are the changes needed?

Fixes an issue in a production environment. Staging directories are being left behind after a materialized view update on an hdfs encrypted table

### Does this PR introduce _any_ user-facing change?

No

### Is the change a dependency upgrade?

No

### How was this patch tested?

Patch was tested in my local environment. Existing materialized view tests should test some of the changes. However, a huge framework change would have to be created to test things are working properly for the materialized view registry test. Given that this is a small change and given that this is a problem on a production environment, I think it's a bit hard to implement this kind of testing in a customer expected amount of time.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org