[jira] [Commented] (HIVE-18942) ALTER TABLE may generate huge event (with all partitions)

["kalyan kumar kalvagadda (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-18942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16396929#comment-16396929 ] 

kalyan kumar kalvagadda commented on HIVE-18942:
------------------------------------------------

[~akolb] That may not be the case.

AddPartitionEvent that is generated here is different from the NotificationEvent that is inserted in the NOTIFICATION_LOG table.

{noformat}
  public void onAlterTable(AlterTableEvent tableEvent) throws MetaException {
    Table before = tableEvent.getOldTable();
    Table after = tableEvent.getNewTable();
    NotificationEvent event =
        new NotificationEvent(0, now(), EventType.ALTER_TABLE.toString(), msgFactory
            .buildAlterTableMessage(before, after, tableEvent.getIsTruncateOp()).toString());
    event.setDbName(after.getDbName());
    event.setTableName(after.getTableName());
    process(event, tableEvent);
  }
{noformat}
 
This is how later table JSON is constructed as sentry is interested in below things in the alter table notification
# Old Db Name
# New Db Name
# Old Table Name
# New Table Name
# Old Location
# New Location

{noformat}
  public JSONAlterTableMessage(String server, String servicePrincipal, Table tableObjBefore, Table tableObjAfter, Long timestamp) {
    this.server = server;
    this.servicePrincipal = servicePrincipal;
    this.db = tableObjBefore.getDbName();
    this.table = tableObjBefore.getTableName();
    this.timestamp = timestamp;

    try {
      this.tableObjBeforeJson = JSONMessageFactory.createTableObjJson(tableObjBefore);
      this.tableObjAfterJson = JSONMessageFactory.createTableObjJson(tableObjAfter);
    } catch (TException var7) {
      throw new IllegalArgumentException("Could not serialize: ", var7);
    }

    this.checkValid();
  }
{noformat}


I think the NotificationEvent inserted in NOTIFICATION_LOG table may not have details of the partitions. 

> ALTER TABLE may generate huge event (with all partitions)
> ---------------------------------------------------------
>
>                 Key: HIVE-18942
>                 URL: https://issues.apache.org/jira/browse/HIVE-18942
>             Project: Hive
>          Issue Type: Bug
>          Components: Metastore
>    Affects Versions: 3.0.0
>            Reporter: Alexander Kolbasov
>            Priority: Major
>
> ALTER TABLE handler in HiveAlterHandler has this code:
> {code:java}
> if (isPartitionedTable) {
>   parts = msdb.getPartitions(newt.getDbName(), newt.getTableName(), -1);
>   MetaStoreListenerNotifier.notifyEvent(transactionalListeners,
>           EventMessage.EventType.ADD_PARTITION,
>           new AddPartitionEvent(newt, parts, true, handler),
>           environmentContext);
> }{code}
> The problem is that table may contain huge number of partitions and the event will contain all of them. Partition object itself isn't very small either, so we may end up with huge events which would be stored and then transmitted over the wire to consumers.
> [~spena] [~kkalyan] [~lina.li] [~vaidyand] FYI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)