You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2020/08/15 02:03:00 UTC
[jira] [Commented] (IMPALA-4364) REFRESH does not pick up ALTER
TABLE...PARTITION...SET LOCATION changes
[ https://issues.apache.org/jira/browse/IMPALA-4364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17178134#comment-17178134 ]
ASF subversion and git services commented on IMPALA-4364:
---------------------------------------------------------
Commit cd52932125e5636ff154c3cdb6a740877b255998 in impala's branch refs/heads/master from Vihang Karajgaonkar
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=cd52932 ]
IMPALA-4364: Query option to refresh updated HMS partitions
This patch introduces a new boolean query option
REFRESH_UPDATED_HMS_PARTITIONS. When this query option is set
the refresh table command reloads the partitions which have been
modified in HMS in addition to adding [removing] the new [removed]
partitions.
In order to do this the refresh table command needs to fetch all
the partitions instead of the just the partition names which can
cause the performance of refresh table to degrade when the query
option is set. However for certain use-cases currently there is
no way to detect changed partitions using refresh table command.
For instance, if certain partition locations have been changed,
a refresh table will not update those partitions.
Testing:
1. Added a new test which sets the query option and makes sure
that the updated partitions from hive are reloaded after refresh
table command.
2. Ran exhaustive tests with the patch.
Change-Id: I50e8680509f4eb0712e7bb3de44df5f2952179af
Reviewed-on: http://gerrit.cloudera.org:8080/16308
Reviewed-by: Quanlong Huang <hu...@gmail.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
> REFRESH does not pick up ALTER TABLE...PARTITION...SET LOCATION changes
> -----------------------------------------------------------------------
>
> Key: IMPALA-4364
> URL: https://issues.apache.org/jira/browse/IMPALA-4364
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Affects Versions: Impala 2.6.0
> Reporter: Jacob Evan Beard
> Assignee: Vihang Karajgaonkar
> Priority: Major
> Labels: usability
>
> AFAIK the REFRESH command should pick up all changes to a table made by ALTER TABLE from outside of Impala (e.g. Spark SQL), however REFRESH does not pick up changes from ALTER TABLE...PARTITION...SET LOCATION, which seems to require an INVALIDATE METADATA instead.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org