[jira] [Comment Edited] (HIVE-19916) master-txnstats branch - integrate with HIVE-19382

["Steve Yeom (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-19916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16514679#comment-16514679 ] 

Steve Yeom edited comment on HIVE-19916 at 6/16/18 7:35 AM:
------------------------------------------------------------

I think I did not clearly mention. Sorry and thank you for your 
great help Sergey. 
 
1. The fact is the write id lists for the query for conf is generated and saved into "conf" 
at acquireLocks() right at the start of execution. 
2. Sergey may be right in saying that the current patch may access the Metastore. I will double
check. 
3. For stats updaters, the current or original patch gets a writeId list from "conf".
The reason why I should take conf-version-writeId-list for a write entity
table is that at acquireLocks() we get a new write id (after +1) for a write entity table of the query.
 
4. But in case of StatsOptimizer I need to get a writeId list for a read entity table and 
use it. But "conf" may not have a writeIdList for the read entity table. 
So I think I have obtained a writeId list for a read entity table based on the global transaction database snapshot that is already taken at the start of query plan optimization. 
But I think this can be the final version for the transaction since I think we can determine commit
level of a table write with this version. I.e., with this version, we can decide open->committed
write instance, open->aborted instance, new open write instance when the 
equivalence method for the patch is invoked, But I will double check. 

P.S.:
I thought the current patch of HIVE-19382 is for checking and regenerating write entity 
table write id lists for I guess UPDATE/DELETE conflict resolution at 
TxnHandler.commitTxns(). Again only for write id lists for write entity tables.
So this does not affect read entity tables.

I may add more after checking..



was (Author: steveyeom2017):
I think I did not clearly mentioned. Sorry Sergey. 

The current or original patch gets conf version writeId list. 
In other words, it uses already taken global transaction snspshot before starting query optimization and get writeIdList based 
on the transaction snapshot like the case of MV and cache result invalidation. The same way!

If needed, I can provide more details.

Thank you Sergey for your great help!


> master-txnstats branch - integrate with HIVE-19382
> --------------------------------------------------
>
>                 Key: HIVE-19916
>                 URL: https://issues.apache.org/jira/browse/HIVE-19916
>             Project: Hive
>          Issue Type: Sub-task
>            Reporter: Sergey Shelukhin
>            Assignee: Steve Yeom
>            Priority: Major
>
> There's some code in original txn stats patch that may go to metastore to get write Ids. This code should not go to metastore, it should fail instead. 
> HIVE-19382 should ensure that we have correct IDs already present during optimizer - they are using by e.g. materialized view optimizer, so they should be there; if they are not present, some integration might be needed so that txn stats optimizations also have access to those write Ids.
> cc [~jcamachorodriguez]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)