[jira] [Commented] (HIVE-18948) Acquire locks before generating the valid transaction list

["Jesus Camacho Rodriguez (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-18948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16408416#comment-16408416 ] 

Jesus Camacho Rodriguez commented on HIVE-18948:
------------------------------------------------

A bit more detail on the work needed to move lock acquisition before the optimization phase.
Call to acquire locks from Driver.java lands in {{acquireLocks}} method in DbTxnManager.
The method takes as parameters {{QueryPlan plan, Context ctx, String username, boolean isBlocking}}. Currently, {{queryPlan}} is not available until query is fully compiled / executed. The following information in the queryPlan is accessed during lock acquisition: query id, operation, read entities, and write entities.
It seems when they query is parsed (after parsing and before optimization), we will already have query id, operation, and read entities.
However, new read entities might be added during compilation/optimization, e.g., when views are used.
In turn, write entities are generated while we are compiling/optimizing the plan though (e.g. when we generate the file sink operator in semantic analyzer), hence we would need to move that logic so it is done while parsing the query (at least we should generate the information that we need). The information that we use from read/write entities is the name of the entity, the type (DATABASE, TABLE, PARTITION, etc.), and sometimes that table object itself (e.g., to check if table is transactional).
Thus, most of the work consists on moving the read entities and write entities generation (or the properties that are needed from them) when we are parsing the query.

> Acquire locks before generating the valid transaction list
> ----------------------------------------------------------
>
>                 Key: HIVE-18948
>                 URL: https://issues.apache.org/jira/browse/HIVE-18948
>             Project: Hive
>          Issue Type: Improvement
>          Components: Transactions
>    Affects Versions: 3.0.0
>            Reporter: Jesus Camacho Rodriguez
>            Priority: Critical
>
> HIVE-18825 moves the valid transaction list generation logic before query optimization. In order to support lock-based concurrency control correctly, the logic to acquire the locks has to be moved before query optimization too, and before the valid transaction list is generated.
> This requires a bit of work/refactoring, since lock acquisition logic relies on read/write entities, and it is heavily dependent on the QueryPlan object too, e.g., it sets some properties in the file sink descriptors for the plan. Currently, all these data structures (except for read entities) are only available after query has been optimized, hence we will need to 1) generate some of this data structures before query optimization, e.g., write entities, and 2) create and propagate some of the properties so they are set during query optimization, e.g., those properties contained in the file sink descriptors.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)