You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2020/06/12 13:13:01 UTC
[jira] [Work logged] (HIVE-19267) Replicate ACID/MM tables write
operations.
[ https://issues.apache.org/jira/browse/HIVE-19267?focusedWorklogId=444858&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444858 ]
ASF GitHub Bot logged work on HIVE-19267:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 12/Jun/20 13:12
Start Date: 12/Jun/20 13:12
Worklog Time Spent: 10m
Work Description: github-actions[bot] commented on pull request #394:
URL: https://github.com/apache/hive/pull/394#issuecomment-643263037
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
Feel free to reach out on the dev@hive.apache.org list if the patch is in need of reviews.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 444858)
Remaining Estimate: 0h
Time Spent: 10m
> Replicate ACID/MM tables write operations.
> ------------------------------------------
>
> Key: HIVE-19267
> URL: https://issues.apache.org/jira/browse/HIVE-19267
> Project: Hive
> Issue Type: Sub-task
> Components: repl, Transactions
> Affects Versions: 3.0.0
> Reporter: mahesh kumar behera
> Assignee: mahesh kumar behera
> Priority: Major
> Labels: ACID, DR, pull-request-available, replication
> Fix For: 4.0.0, 3.2.0
>
> Attachments: HIVE-19267.01-branch-3.patch, HIVE-19267.01.patch, HIVE-19267.02-branch-3.patch, HIVE-19267.02.patch, HIVE-19267.03.patch, HIVE-19267.04.patch, HIVE-19267.05.patch, HIVE-19267.06.patch, HIVE-19267.07.patch, HIVE-19267.08.patch, HIVE-19267.09.patch, HIVE-19267.10.patch, HIVE-19267.11.patch, HIVE-19267.12.patch, HIVE-19267.13.patch, HIVE-19267.14.patch, HIVE-19267.15.patch, HIVE-19267.16.patch, HIVE-19267.17.patch, HIVE-19267.18.patch, HIVE-19267.19.patch, HIVE-19267.20.patch, HIVE-19267.21.patch, HIVE-19267.22.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
>
> h1. Replicate ACID write Events
> * Create new EVENT_WRITE event with related message format to log the write operations with in a txn along with data associated.
> * Log this event when perform any writes (insert into, insert overwrite, load table, delete, update, merge, truncate) on table/partition.
> * If a single MERGE/UPDATE/INSERT/DELETE statement operates on multiple partitions, then need to log one event per partition.
> * DbNotificationListener should log this type of event to special metastore table named "MTxnWriteNotificationLog".
> * This table should maintain a map of txn ID against list of tables/partitions written by given txn.
> * The entry for a given txn should be removed by the cleaner thread that removes the expired events from EventNotificationTable.
> h1. Replicate Commit Txn operation (with writes)
> Add new EVENT_COMMIT_TXN to log the metadata/data of all tables/partitions modified within the txn.
> *Source warehouse:*
> * This event should read the EVENT_WRITEs from "MTxnWriteNotificationLog" metastore table to consolidate the list of tables/partitions modified within this txn scope.
> * Based on the list of tables/partitions modified and table Write ID, need to compute the list of delta files added by this txn.
> * Repl dump should read this message and dump the metadata and delta files list.
> *Target warehouse:*
> * Ensure snapshot isolation at target for on-going read txns which shouldn't view the data replicated from committed txn. (Ensured with open and allocate write ID events).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)