You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Karthik Manamcheri (JIRA)" <ji...@apache.org> on 2019/01/02 17:27:00 UTC
[jira] [Updated] (HIVE-20977) Lazy evaluate the table object in
PreReadTableEvent to improve get_partition performance
[ https://issues.apache.org/jira/browse/HIVE-20977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Karthik Manamcheri updated HIVE-20977:
--------------------------------------
Attachment: HIVE-20977.2.patch
> Lazy evaluate the table object in PreReadTableEvent to improve get_partition performance
> ----------------------------------------------------------------------------------------
>
> Key: HIVE-20977
> URL: https://issues.apache.org/jira/browse/HIVE-20977
> Project: Hive
> Issue Type: Improvement
> Reporter: Karthik Manamcheri
> Assignee: Karthik Manamcheri
> Priority: Minor
> Attachments: HIVE-20977.1.patch, HIVE-20977.2.patch
>
>
> The PreReadTableEvent is generated for non-table operations (such as get_partitions), but only if there is an event listener attached. However, this is also not necessary if the event listener is not interested in the read table event.
> For example, the TransactionalValidationListener's onEvent looks like this
> {code:java}
> @Override
> public void onEvent(PreEventContext context) throws MetaException, NoSuchObjectException,
> InvalidOperationException {
> switch (context.getEventType()) {
> case CREATE_TABLE:
> handle((PreCreateTableEvent) context);
> break;
> case ALTER_TABLE:
> handle((PreAlterTableEvent) context);
> break;
> default:
> //no validation required..
> }
> }{code}
>
> Note that for read table events it is a no-op. The problem is that the get_table is evaluated when creating the PreReadTableEvent finally to be just ignored!
> Look at the code below.. {{getMS().getTable(..)}} is evaluated irrespective of if the listener uses it or not.
> {code:java}
> private void fireReadTablePreEvent(String catName, String dbName, String tblName)
> throws MetaException, NoSuchObjectException {
> if(preListeners.size() > 0) {
> // do this only if there is a pre event listener registered (avoid unnecessary
> // metastore api call)
> Table t = getMS().getTable(catName, dbName, tblName);
> if (t == null) {
> throw new NoSuchObjectException(TableName.getQualified(catName, dbName, tblName)
> + " table not found");
> }
> firePreEvent(new PreReadTableEvent(t, this));
> }
> }
> {code}
> This can be improved by using a {{Supplier}} and lazily evaluating the table when needed (once when the first time it is called, memorized after that).
> *Motivation*
> Whenever a partition call occurs (get_partition, etc.), we fire the PreReadTableEvent. This affects performance since it fetches the table even if it is not being used. This change will improve performance on the get_partition calls.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)