You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@hive.apache.org by "Eugene Koifman (JIRA)" <ji...@apache.org> on 2016/03/31 23:11:25 UTC

[jira] [Created] (HIVE-13403) Make Streaming API not create empty buckets (at least as an option)

Eugene Koifman created HIVE-13403:
-------------------------------------

             Summary: Make Streaming API not create empty buckets (at least as an option)
                 Key: HIVE-13403
                 URL: https://issues.apache.org/jira/browse/HIVE-13403
             Project: Hive
          Issue Type: Bug
          Components: HCatalog, Transactions
    Affects Versions: 1.3.0
            Reporter: Eugene Koifman
            Assignee: Eugene Koifman


as of HIVE-11983, when a TransactionBatch is opened in StreamingAPI, a full compliment of bucket files is created on disk even though some may end up receiving no data.

It would be better to create them on demand and not clog the FS.

Tez can handle missing (empty) buckets and on MR bucket join algorithms will check if all buckets are there and bail out if not.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)