[jira] [Created] (HIVE-11987) CompactionTxnHandler.createValidCompactTxnList() can use much less memory

["Eugene Koifman (JIRA)" <ji...@apache.org>]

Eugene Koifman created HIVE-11987:
-------------------------------------

             Summary: CompactionTxnHandler.createValidCompactTxnList() can use much less memory
                 Key: HIVE-11987
                 URL: https://issues.apache.org/jira/browse/HIVE-11987
             Project: Hive
          Issue Type: Bug
          Components: Metastore, Transactions
    Affects Versions: 1.1.0
            Reporter: Eugene Koifman
            Assignee: Eugene Koifman
            Priority: Minor


This method only needs HWM and list of txn IDs in 'o' state.

It's currently implemented to get the list from TxnHandler.getOpenTxnsInfo(),
which returns (txn id, state, host, user) for each txn and includes Aborted txns.

This can easily be 120 bytes or more per txn overhead (over 1 Java long) which not an issue in general but when the system is misconfigured, the number of opened/aborted txns can get into the millions.  This creates unnecessary memory pressure on metastore.

Should consider fixing this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)