[jira] [Updated] (HIVE-21158) Perform update split early

["Eugene Koifman (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HIVE-21158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eugene Koifman updated HIVE-21158:
----------------------------------
    Description: 
Currently Acid 2.0 does U=D+I in the OrcRecordUpdater. This means that all Updates (wide rows) are shuffled AND sorted.
 We could modify the the multi-insert statement which results from Merge statement so that instead of having one of the legs represent Update, we create 2 legs - 1 representing Delete of original row and 1 representing Insert of the new version.
 Delete events are very small so sorting them is cheap. The Insert are written to disk in a sorted way by virtue of how ROW__IDs are generated.

Exactly the same idea applies to regular Update statement.

Note that the U=D+I in OrcRecordUpdater needs to be kept to keep [Streaming Mutate API |https://cwiki.apache.org/confluence/display/Hive/HCatalog+Streaming+Mutation+API] working on 2.0.

*This requires that TxnHandler flags 2 Deletes as a conflict - it doesn't currently*

Incidentally, 2.0 + early split allows updating all columns including bucketing and partition columns

What is lock acquisition based on? Need to make sure that conflict detection (write set tracking) still works

So we want to transform
{noformat}
update T set B = 7 where A=1
{noformat}
into
{noformat}
from T
insert into T select ROW__ID where a = 1 SORT BY ROW__ID
insert into T select a, 7 where a = 1
{noformat}
even better to
{noformat}
from T where a = 1
insert into T select ROW__ID SORT BY ROW__ID
insert into T select a, 7
{noformat}
but this won't parse currently.

This is very similar to how MERGE stmt is handled.

Need some though on on how WriteSet tracking works. If we don't allow updating partition column, then even with dynamic partitions TxnHandler.addDynamicPartitions() should see 1 entry (in Update type) for each partition since both the insert and delete land in the same partition. If part cols can be updated, then then we may insert a Delete event into P1 and corresponding Insert event into P2 so addDynamicPartitions() should see both parts. I guess both need to be recored in Write_Set but with different types. The delete as 'delete' and insert as insert so that it can conflict with some IOW on the 'new' partition.

  was:
Currently Acid 2.0 does U=D+I in the OrcRecordUpdater.  This means that all Updates (wide rows) are shuffled AND sorted.
We could modify the the multi-insert statement which results from Merge statement so that instead of having one of the legs represent Update, we create 2 legs - 1 representing Delete of original row and 1 representing Insert of the new version.
Delete events are very small so sorting them is cheap.  The Insert are written to disk in a sorted way by virtue of how ROW__IDs are generated.

Exactly the same idea applies to regular Update statement.

Note that the U=D+I in OrcRecordUpdater needs to be kept to keep [Streaming Mutate API |https://cwiki.apache.org/confluence/display/Hive/HCatalog+Streaming+Mutation+API] working on 2.0.

*This requires that TxnHandler flags 2 Deletes as a conflict - it doesn't currently*

Incidentally, 2.0 + early split allows updating all columns including bucketing and partition columns

What is lock acquisition based on?  Need to make sure that conflict detection (write set tracking) still works

So we want to transform
{noformat}
update T set B = 7 where A=1
{noformat}
into 
{noformat}
from T
insert into T select ROW__ID where a = 1 SORT BY ROW__ID
insert into T select a, 7 where a = 1
{noformat}

even better to
{noformat}
from T where a = 1
insert into T select ROW__ID SORT BY ROW__ID
insert into T select a, 7
{noformat}
but this won't parse currently.

This is very similar to how MERGE stmt is handled.

Need some though on on how WriteSet tracking works.  If we don't allow updating partition column, then even with dynamic partitions TxnHandler.addDynamicPartitions() should see 1 entry (in Update type) for each partition since both the insert and delete land in the same partition.  If part cols can be updated, then then we may insert a Delete event into P1 and corresponding Insert event into P2 so addDynamicPartitions() should see both parts.  I guess both need to be recored in Write_Set but with different types.  The delete as 'update' and insert as insert so that it can conflict with some IOW on the 'new' partition.


> Perform update split early
> --------------------------
>
>                 Key: HIVE-21158
>                 URL: https://issues.apache.org/jira/browse/HIVE-21158
>             Project: Hive
>          Issue Type: Improvement
>          Components: Transactions
>    Affects Versions: 3.0.0
>            Reporter: Eugene Koifman
>            Assignee: Eugene Koifman
>            Priority: Major
>
> Currently Acid 2.0 does U=D+I in the OrcRecordUpdater. This means that all Updates (wide rows) are shuffled AND sorted.
>  We could modify the the multi-insert statement which results from Merge statement so that instead of having one of the legs represent Update, we create 2 legs - 1 representing Delete of original row and 1 representing Insert of the new version.
>  Delete events are very small so sorting them is cheap. The Insert are written to disk in a sorted way by virtue of how ROW__IDs are generated.
> Exactly the same idea applies to regular Update statement.
> Note that the U=D+I in OrcRecordUpdater needs to be kept to keep [Streaming Mutate API |https://cwiki.apache.org/confluence/display/Hive/HCatalog+Streaming+Mutation+API] working on 2.0.
> *This requires that TxnHandler flags 2 Deletes as a conflict - it doesn't currently*
> Incidentally, 2.0 + early split allows updating all columns including bucketing and partition columns
> What is lock acquisition based on? Need to make sure that conflict detection (write set tracking) still works
> So we want to transform
> {noformat}
> update T set B = 7 where A=1
> {noformat}
> into
> {noformat}
> from T
> insert into T select ROW__ID where a = 1 SORT BY ROW__ID
> insert into T select a, 7 where a = 1
> {noformat}
> even better to
> {noformat}
> from T where a = 1
> insert into T select ROW__ID SORT BY ROW__ID
> insert into T select a, 7
> {noformat}
> but this won't parse currently.
> This is very similar to how MERGE stmt is handled.
> Need some though on on how WriteSet tracking works. If we don't allow updating partition column, then even with dynamic partitions TxnHandler.addDynamicPartitions() should see 1 entry (in Update type) for each partition since both the insert and delete land in the same partition. If part cols can be updated, then then we may insert a Delete event into P1 and corresponding Insert event into P2 so addDynamicPartitions() should see both parts. I guess both need to be recored in Write_Set but with different types. The delete as 'delete' and insert as insert so that it can conflict with some IOW on the 'new' partition.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)