[jira] [Created] (HIVE-27018) Aborted transaction cleanup outside compaction process

["Sourabh Badhya (Jira)" <ji...@apache.org>]

Sourabh Badhya created HIVE-27018:
-------------------------------------

             Summary:  Aborted transaction cleanup outside compaction process
                 Key: HIVE-27018
                 URL: https://issues.apache.org/jira/browse/HIVE-27018
             Project: Hive
          Issue Type: Improvement
            Reporter: Sourabh Badhya
            Assignee: Sourabh Badhya


Aborted transactions processing is tightly integrated into the compaction pipeline and consists of 3 main stages: Initiator, Compactor (Worker), Cleaner. That could be simplified by doing all work on the Cleaner side.


*Potential Benefits -* 
There are major advantages of implementing this on the cleaner side - 
 1) Currently an aborted txn in the TXNS table blocks the cleaning of TXN_TO_WRITE_ID table since nothing gets cleaned above MIN(aborted txnid) in the current implementation. After implementing this on the cleaner side, the cleaner regularly checks and cleans the aborted records in the TXN_COMPONENTS table, which in turn makes the AcidTxnCleanerService clean the aborted txns in TXNS table.
 2) Initiator and worker do not do anything on tables which contain only aborted directories. It's the cleaner which removes the aborted directories of the table. Hence all operations associated with the initiator and worker for these tables are wasteful. These wasteful operations are avoided.
3) DP writes which are aborted are skipped by the worker currently. Hence once again the cleaner is the one deleting the aborted directories. All operations associated with the initiator and worker for this entry are wasteful. These wasteful operations are avoided.

*Proposed solution -* 
*Implement logic to handle aborted transactions exclusively in Cleaner.*
Implement logic to fetch the TXN_COMPONENTS which are associated with transactions in aborted state and send the required information to the cleaner. Cleaner must clean up the aborted deltas/delete deltas by using the aborted directories in the AcidState of the table/partition.
It is also better to separate entities which provide information of compaction and abort cleanup to enhance code modularity. This can be done in this way -


Cleaner can be divided into separate entities like - 
*1) Handler* - This entity fetches the data from the metastore DB from relevant tables and converts it into a request entity called CleaningRequest. It would also do SQL operations post cleanup (postprocess). Every type of cleaning request is provided by a separate handler.
*2) Filesystem remover* - This entity fetches the cleaning requests from various handlers and deletes them according to the cleaning request.

*This division allows for dynamic extensibility of cleanup from multiple handlers. Every handler is responsible for providing cleaning requests from a specific source.*

The following solution is resilient i.e. in the event of abrupt metastore shutdown, the cleaner can still see the relevant entries in the metastore DB and retry the cleaning task for that entry.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)