[jira] [Updated] (HIVE-21146) Enforce TransactionBatch size=1 for blob stores

["David Lavati (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HIVE-21146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

David Lavati updated HIVE-21146:
--------------------------------
    Attachment: HIVE-21146.2.patch

> Enforce TransactionBatch size=1 for blob stores
> -----------------------------------------------
>
>                 Key: HIVE-21146
>                 URL: https://issues.apache.org/jira/browse/HIVE-21146
>             Project: Hive
>          Issue Type: Bug
>          Components: Streaming, Transactions
>    Affects Versions: 3.0.0
>            Reporter: Eugene Koifman
>            Assignee: David Lavati
>            Priority: Major
>              Labels: pull-request-available
>         Attachments: HIVE-21146.2.patch, HIVE-21146.patch
>
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> Streaming Ingest API supports a concept of {{TransactionBatch}} where N transactions can be opened at once and the data in all of them will be written to the same delta_x_y directory where each transaction in the batch can be committed/aborted independently.  The implementation relies on {{FSDataOutputStream.hflush()}} (called from OrcRecordUpdater}} which is available on HDFS but is often implemented as no-op in Blob store backed {{FileSystem}} objects.
> Need to add a check to {{HiveStreamingConnection()}} constructor to raise an error if {{builder.transactionBatchSize > 1}} and the target table/partitions are backed by something that doesn't support {{hflush()}}.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)