You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Manoj Murumkar <ma...@gmail.com> on 2016/11/17 01:24:20 UTC
Hive transactions not working
Hi,
We are trying to implement transaction feature in hive. I created following
table:
+----------------------------------------------------------------------------------------+--+
|
createtab_stmt |
+----------------------------------------------------------------------------------------+--+
| CREATE TABLE
`txntest.txntab3`( |
| `id`
int,
|
| `name`
string)
|
| CLUSTERED BY
( |
|
id)
|
| INTO 2
BUCKETS
|
| ROW FORMAT
SERDE
|
|
'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
|
| STORED AS
INPUTFORMAT
|
|
'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
|
|
OUTPUTFORMAT
|
|
'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
|
|
LOCATION
|
| 'hdfs://
or1010051016175.corp.adobe.com:8020/user/hive/warehouse/txntest.db/txntab3'
|
| TBLPROPERTIES
( |
|
'COLUMN_STATS_ACCURATE'='true',
|
|
'numFiles'='22',
|
|
'numRows'='90000',
|
|
'rawDataSize'='0',
|
|
'totalSize'='3564019',
|
|
'transactional'='true',
|
|
'transient_lastDdlTime'='1479329198')
|
+----------------------------------------------------------------------------------------+--+
I inserted 90000 rows in it in multiple iterations, so it created 22 files
(as is visible above). I have run multiple compactions (major and minor),
but nothing seems to happen on HDFS. What am I missing?
I have following configuration:
Metastore:
hive.compactor.initiator.on = true;
hive.compactor.worker.threads = 2;
Client:
hive.support.concurrency = true;
hive.exec.dynamic.partition.mode = nonstrict;
hive.txn.manager = org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
Can someone point me in right direction? Compaction process did run
(verified via SHOW COMPACTIONS) and I also see we have base directory
created on HDFS.
I was expecting all the delta directories gone when major compation runs.
drwxrwxrwt - admin hive 0 2016-11-16 20:47
/user/hive/warehouse/txntest.db/txntab3/base_0000021
-rw-r--r-- 3 admin hive 227916 2016-11-16 20:47
/user/hive/warehouse/txntest.db/txntab3/base_0000021/bucket_00000
drwxr-xr-x - nex37045 hive 0 2016-11-16 01:33
/user/hive/warehouse/txntest.db/txntab3/delta_0000003_0000003
-rw-r--r-- 3 nex37045 hive 640 2016-11-16 01:33
/user/hive/warehouse/txntest.db/txntab3/delta_0000003_0000003/bucket_00000
drwxr-xr-x - nex37045 hive 0 2016-11-16 01:33
/user/hive/warehouse/txntest.db/txntab3/delta_0000004_0000004
-rw-r--r-- 3 nex37045 hive 640 2016-11-16 01:33
/user/hive/warehouse/txntest.db/txntab3/delta_0000004_0000004/bucket_00000
drwxr-xr-x - nex37045 hive 0 2016-11-16 01:34
/user/hive/warehouse/txntest.db/txntab3/delta_0000005_0000005
-rw-r--r-- 3 nex37045 hive 640 2016-11-16 01:33
/user/hive/warehouse/txntest.db/txntab3/delta_0000005_0000005/bucket_00000
drwxr-xr-x - nex37045 hive 0 2016-11-16 01:34
/user/hive/warehouse/txntest.db/txntab3/delta_0000006_0000006
-rw-r--r-- 3 nex37045 hive 640 2016-11-16 01:34
/user/hive/warehouse/txntest.db/txntab3/delta_0000006_0000006/bucket_00000
drwxr-xr-x - nex37045 hive 0 2016-11-16 01:34
/user/hive/warehouse/txntest.db/txntab3/delta_0000007_0000007
-rw-r--r-- 3 nex37045 hive 636 2016-11-16 01:34
/user/hive/warehouse/txntest.db/txntab3/delta_0000007_0000007/bucket_00000
drwxr-xr-x - nex37045 hive 0 2016-11-16 01:36
/user/hive/warehouse/txntest.db/txntab3/delta_0000008_0000008
-rw-r--r-- 3 nex37045 hive 640 2016-11-16 01:36
/user/hive/warehouse/txntest.db/txntab3/delta_0000008_0000008/bucket_00000
drwxr-xr-x - nex37045 hive 0 2016-11-16 01:36
/user/hive/warehouse/txntest.db/txntab3/delta_0000009_0000009
-rw-r--r-- 3 nex37045 hive 640 2016-11-16 01:36
/user/hive/warehouse/txntest.db/txntab3/delta_0000009_0000009/bucket_00000
drwxr-xr-x - nex37045 hive 0 2016-11-16 01:37
/user/hive/warehouse/txntest.db/txntab3/delta_0000010_0000010
-rw-r--r-- 3 nex37045 hive 640 2016-11-16 01:37
/user/hive/warehouse/txntest.db/txntab3/delta_0000010_0000010/bucket_00000
drwxr-xr-x - nex37045 hive 0 2016-11-16 01:37
/user/hive/warehouse/txntest.db/txntab3/delta_0000011_0000011
-rw-r--r-- 3 nex37045 hive 644 2016-11-16 01:37
/user/hive/warehouse/txntest.db/txntab3/delta_0000011_0000011/bucket_00000
drwxr-xr-x - nex37045 hive 0 2016-11-16 01:37
/user/hive/warehouse/txntest.db/txntab3/delta_0000012_0000012
-rw-r--r-- 3 nex37045 hive 644 2016-11-16 01:37
/user/hive/warehouse/txntest.db/txntab3/delta_0000012_0000012/bucket_00000
drwxr-xr-x - nex37045 hive 0 2016-11-16 01:44
/user/hive/warehouse/txntest.db/txntab3/delta_0000013_0000013
-rw-r--r-- 3 nex37045 hive 644 2016-11-16 01:44
/user/hive/warehouse/txntest.db/txntab3/delta_0000013_0000013/bucket_00000
drwxr-xr-x - nex37045 hive 0 2016-11-16 01:45
/user/hive/warehouse/txntest.db/txntab3/delta_0000014_0000014
-rw-r--r-- 3 nex37045 hive 644 2016-11-16 01:45
/user/hive/warehouse/txntest.db/txntab3/delta_0000014_0000014/bucket_00000
drwxr-xr-x - nex37045 hive 0 2016-11-16 02:02
/user/hive/warehouse/txntest.db/txntab3/delta_0000015_0000015
-rw-r--r-- 3 nex37045 hive 644 2016-11-16 02:02
/user/hive/warehouse/txntest.db/txntab3/delta_0000015_0000015/bucket_00000
drwxrwxrwt - admin hive 0 2016-11-16 02:03
/user/hive/warehouse/txntest.db/txntab3/delta_0000015_0000016
-rw-r--r-- 3 admin hive 531 2016-11-16 02:03
/user/hive/warehouse/txntest.db/txntab3/delta_0000015_0000016/bucket_00000
drwxr-xr-x - nex37045 hive 0 2016-11-16 02:03
/user/hive/warehouse/txntest.db/txntab3/delta_0000016_0000016
-rw-r--r-- 3 nex37045 hive 644 2016-11-16 02:03
/user/hive/warehouse/txntest.db/txntab3/delta_0000016_0000016/bucket_00000
drwxr-xr-x - nex37045 hive 0 2016-11-16 20:37
/user/hive/warehouse/txntest.db/txntab3/delta_0000017_0000017
-rw-r--r-- 3 nex37045 hive 156395 2016-11-16 20:37
/user/hive/warehouse/txntest.db/txntab3/delta_0000017_0000017/bucket_00000
drwxrwxrwt - admin hive 0 2016-11-16 20:40
/user/hive/warehouse/txntest.db/txntab3/delta_0000017_0000019
-rw-r--r-- 3 admin hive 2598250 2016-11-16 20:40
/user/hive/warehouse/txntest.db/txntab3/delta_0000017_0000019/bucket_00000
drwxr-xr-x - nex37045 hive 0 2016-11-16 20:39
/user/hive/warehouse/txntest.db/txntab3/delta_0000018_0000018
-rw-r--r-- 3 nex37045 hive 4737 2016-11-16 20:39
/user/hive/warehouse/txntest.db/txntab3/delta_0000018_0000018/bucket_00000
drwxr-xr-x - nex37045 hive 0 2016-11-16 20:39
/user/hive/warehouse/txntest.db/txntab3/delta_0000019_0000019
-rw-r--r-- 3 nex37045 hive 192658 2016-11-16 20:39
/user/hive/warehouse/txntest.db/txntab3/delta_0000019_0000019/bucket_00000
drwxr-xr-x - nex37045 hive 0 2016-11-16 20:45
/user/hive/warehouse/txntest.db/txntab3/delta_0000020_0000020
-rw-r--r-- 3 nex37045 hive 192835 2016-11-16 20:45
/user/hive/warehouse/txntest.db/txntab3/delta_0000020_0000020/bucket_00000
drwxr-xr-x - nex37045 hive 0 2016-11-16 20:46
/user/hive/warehouse/txntest.db/txntab3/delta_0000021_0000021
-rw-r--r-- 3 nex37045 hive 201206 2016-11-16 20:46
/user/hive/warehouse/txntest.db/txntab3/delta_0000021_0000021/bucket_00000
Thanks,
Manoj
Re: Hive transactions not working
Posted by Manoj Murumkar <ma...@gmail.com>.
By default, I have these:
hadoop.proxyuser.hive.hosts=*
hadoop.proxyuser.hive.groups=*
Also, I have impersonation enabled. These settings should have made the
compaction to run without issues, but does not. What am I missing?
On Wed, Nov 16, 2016 at 6:00 PM, Manoj Murumkar <ma...@gmail.com>
wrote:
> Okay, made the cleaner work as well :)
>
> When the files are owned by a user that is different from the user that
> runs metastore ("hive"), this doesn't work. Just to test this, I changed
> ownership of the files for the table to "hive" and ran a major compaction
> and it does the job.
>
> drwxrwxrwt - hive hive 0 2016-11-17 01:35
> /user/hive/warehouse/txntest.db/txntab3/base_0000024
> -rw-r--r-- 3 hive hive 451008 2016-11-17 01:35
> /user/hive/warehouse/txntest.db/txntab3/base_0000024/bucket_00000
>
> I am going to follow this piece of documentation, so it works:
> Configuration Values to Set for Compaction
>
> If the data in your system is not owned by the Hive user (i.e., the user
> that the Hive metastore runs as), then Hive will need permission to run as
> the user who owns the data in order to perform compactions. If you have
> already set up HiveServer2 to impersonate users, then the only additional
> work to do is assure that Hive has the right to impersonate users from the
> host running the Hive metastore. This is done by adding the hostname to
> hadoop.proxyuser.hive.hosts in Hadoop's core-site.xml file. If you have
> not already done this, then you will need to configure Hive to act as a
> proxy user. This requires you to set up keytabs for the user running the
> Hive metastore and add hadoop.proxyuser.hive.hosts and hadoop.proxyuser.
> hive.groups to Hadoop's core-site.xml file. See the Hadoop documentation
> on secure mode for your version of Hadoop (e.g., for Hadoop 2.5.1 it is at Hadoop
> in Secure Mode
> <http://hadoop.apache.org/docs/r2.5.1/hadoop-project-dist/hadoop-common/SecureMode.html>
> ).
>
>
> On Wed, Nov 16, 2016 at 5:43 PM, Manoj Murumkar <ma...@gmail.com>
> wrote:
>
>> Further update:
>>
>> I see this:
>>
>> +-----------+----------+------------+--------+--------------
>> -------+-----------+----------------+--+
>> | dbname | tabname | partname | type | state |
>> workerid | starttime |
>> +-----------+----------+------------+--------+--------------
>> -------+-----------+----------------+--+
>> | Database | Table | Partition | Type | State |
>> Worker | Start Time |
>> | txntest | txntab3 | NULL | MAJOR | ready for cleaning |
>> NULL | 1479346924000 |
>> +-----------+----------+------------+--------+--------------
>> -------+-----------+----------------+--+
>>
>> However, I don't see cleaner actually run in hive logs (I am looking for
>> a string like "compactor.Cleaner").
>>
>>
>> On Wed, Nov 16, 2016 at 5:30 PM, Manoj Murumkar <manoj.murumkar@gmail.com
>> > wrote:
>>
>>> Quick update:
>>>
>>> After each compaction, files under base directory (for the buckets) have
>>> latest data. However, I am expecting to see all delta files (and
>>> directories) gone, as they should be merged in the base directory.
>>> Otherwise, we'll start seeing too many small files on HDFS which is a
>>> problem. Am I understanding this feature correctly in assuming so?
>>>
>>> On Wed, Nov 16, 2016 at 5:24 PM, Manoj Murumkar <
>>> manoj.murumkar@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> We are trying to implement transaction feature in hive. I created
>>>> following table:
>>>>
>>>> +-----------------------------------------------------------
>>>> -----------------------------+--+
>>>> | createtab_stmt
>>>> |
>>>> +-----------------------------------------------------------
>>>> -----------------------------+--+
>>>> | CREATE TABLE `txntest.txntab3`(
>>>> |
>>>> | `id` int,
>>>> |
>>>> | `name` string)
>>>> |
>>>> | CLUSTERED BY (
>>>> |
>>>> | id)
>>>> |
>>>> | INTO 2 BUCKETS
>>>> |
>>>> | ROW FORMAT SERDE
>>>> |
>>>> | 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
>>>> |
>>>> | STORED AS INPUTFORMAT
>>>> |
>>>> | 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
>>>> |
>>>> | OUTPUTFORMAT
>>>> |
>>>> | 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
>>>> |
>>>> | LOCATION
>>>> |
>>>> | 'hdfs://or1010051016175.corp.adobe.com:8020/user/hive/wareho
>>>> use/txntest.db/txntab3' |
>>>> | TBLPROPERTIES (
>>>> |
>>>> | 'COLUMN_STATS_ACCURATE'='true',
>>>> |
>>>> | 'numFiles'='22',
>>>> |
>>>> | 'numRows'='90000',
>>>> |
>>>> | 'rawDataSize'='0',
>>>> |
>>>> | 'totalSize'='3564019',
>>>> |
>>>> | 'transactional'='true',
>>>> |
>>>> | 'transient_lastDdlTime'='1479329198')
>>>> |
>>>> +-----------------------------------------------------------
>>>> -----------------------------+--+
>>>>
>>>> I inserted 90000 rows in it in multiple iterations, so it created 22
>>>> files (as is visible above). I have run multiple compactions (major and
>>>> minor), but nothing seems to happen on HDFS. What am I missing?
>>>>
>>>> I have following configuration:
>>>>
>>>> Metastore:
>>>>
>>>> hive.compactor.initiator.on = true;
>>>> hive.compactor.worker.threads = 2;
>>>>
>>>> Client:
>>>>
>>>> hive.support.concurrency = true;
>>>> hive.exec.dynamic.partition.mode = nonstrict;
>>>> hive.txn.manager = org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
>>>>
>>>> Can someone point me in right direction? Compaction process did run
>>>> (verified via SHOW COMPACTIONS) and I also see we have base directory
>>>> created on HDFS.
>>>> I was expecting all the delta directories gone when major compation
>>>> runs.
>>>>
>>>> drwxrwxrwt - admin hive 0 2016-11-16 20:47
>>>> /user/hive/warehouse/txntest.db/txntab3/base_0000021
>>>> -rw-r--r-- 3 admin hive 227916 2016-11-16 20:47
>>>> /user/hive/warehouse/txntest.db/txntab3/base_0000021/bucket_00000
>>>> drwxr-xr-x - nex37045 hive 0 2016-11-16 01:33
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000003_0000003
>>>> -rw-r--r-- 3 nex37045 hive 640 2016-11-16 01:33
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000003_000000
>>>> 3/bucket_00000
>>>> drwxr-xr-x - nex37045 hive 0 2016-11-16 01:33
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000004_0000004
>>>> -rw-r--r-- 3 nex37045 hive 640 2016-11-16 01:33
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000004_000000
>>>> 4/bucket_00000
>>>> drwxr-xr-x - nex37045 hive 0 2016-11-16 01:34
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000005_0000005
>>>> -rw-r--r-- 3 nex37045 hive 640 2016-11-16 01:33
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000005_000000
>>>> 5/bucket_00000
>>>> drwxr-xr-x - nex37045 hive 0 2016-11-16 01:34
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000006_0000006
>>>> -rw-r--r-- 3 nex37045 hive 640 2016-11-16 01:34
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000006_000000
>>>> 6/bucket_00000
>>>> drwxr-xr-x - nex37045 hive 0 2016-11-16 01:34
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000007_0000007
>>>> -rw-r--r-- 3 nex37045 hive 636 2016-11-16 01:34
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000007_000000
>>>> 7/bucket_00000
>>>> drwxr-xr-x - nex37045 hive 0 2016-11-16 01:36
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000008_0000008
>>>> -rw-r--r-- 3 nex37045 hive 640 2016-11-16 01:36
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000008_000000
>>>> 8/bucket_00000
>>>> drwxr-xr-x - nex37045 hive 0 2016-11-16 01:36
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000009_0000009
>>>> -rw-r--r-- 3 nex37045 hive 640 2016-11-16 01:36
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000009_000000
>>>> 9/bucket_00000
>>>> drwxr-xr-x - nex37045 hive 0 2016-11-16 01:37
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000010_0000010
>>>> -rw-r--r-- 3 nex37045 hive 640 2016-11-16 01:37
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000010_000001
>>>> 0/bucket_00000
>>>> drwxr-xr-x - nex37045 hive 0 2016-11-16 01:37
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000011_0000011
>>>> -rw-r--r-- 3 nex37045 hive 644 2016-11-16 01:37
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000011_000001
>>>> 1/bucket_00000
>>>> drwxr-xr-x - nex37045 hive 0 2016-11-16 01:37
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000012_0000012
>>>> -rw-r--r-- 3 nex37045 hive 644 2016-11-16 01:37
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000012_000001
>>>> 2/bucket_00000
>>>> drwxr-xr-x - nex37045 hive 0 2016-11-16 01:44
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000013_0000013
>>>> -rw-r--r-- 3 nex37045 hive 644 2016-11-16 01:44
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000013_000001
>>>> 3/bucket_00000
>>>> drwxr-xr-x - nex37045 hive 0 2016-11-16 01:45
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000014_0000014
>>>> -rw-r--r-- 3 nex37045 hive 644 2016-11-16 01:45
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000014_000001
>>>> 4/bucket_00000
>>>> drwxr-xr-x - nex37045 hive 0 2016-11-16 02:02
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000015_0000015
>>>> -rw-r--r-- 3 nex37045 hive 644 2016-11-16 02:02
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000015_000001
>>>> 5/bucket_00000
>>>> drwxrwxrwt - admin hive 0 2016-11-16 02:03
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000015_0000016
>>>> -rw-r--r-- 3 admin hive 531 2016-11-16 02:03
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000015_000001
>>>> 6/bucket_00000
>>>> drwxr-xr-x - nex37045 hive 0 2016-11-16 02:03
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000016_0000016
>>>> -rw-r--r-- 3 nex37045 hive 644 2016-11-16 02:03
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000016_000001
>>>> 6/bucket_00000
>>>> drwxr-xr-x - nex37045 hive 0 2016-11-16 20:37
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000017_0000017
>>>> -rw-r--r-- 3 nex37045 hive 156395 2016-11-16 20:37
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000017_000001
>>>> 7/bucket_00000
>>>> drwxrwxrwt - admin hive 0 2016-11-16 20:40
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000017_0000019
>>>> -rw-r--r-- 3 admin hive 2598250 2016-11-16 20:40
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000017_000001
>>>> 9/bucket_00000
>>>> drwxr-xr-x - nex37045 hive 0 2016-11-16 20:39
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000018_0000018
>>>> -rw-r--r-- 3 nex37045 hive 4737 2016-11-16 20:39
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000018_000001
>>>> 8/bucket_00000
>>>> drwxr-xr-x - nex37045 hive 0 2016-11-16 20:39
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000019_0000019
>>>> -rw-r--r-- 3 nex37045 hive 192658 2016-11-16 20:39
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000019_000001
>>>> 9/bucket_00000
>>>> drwxr-xr-x - nex37045 hive 0 2016-11-16 20:45
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000020_0000020
>>>> -rw-r--r-- 3 nex37045 hive 192835 2016-11-16 20:45
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000020_000002
>>>> 0/bucket_00000
>>>> drwxr-xr-x - nex37045 hive 0 2016-11-16 20:46
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000021_0000021
>>>> -rw-r--r-- 3 nex37045 hive 201206 2016-11-16 20:46
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000021_000002
>>>> 1/bucket_00000
>>>>
>>>>
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> Manoj
>>>>
>>>>
>>>
>>
>
Re: Hive transactions not working
Posted by Manoj Murumkar <ma...@gmail.com>.
Okay, made the cleaner work as well :)
When the files are owned by a user that is different from the user that
runs metastore ("hive"), this doesn't work. Just to test this, I changed
ownership of the files for the table to "hive" and ran a major compaction
and it does the job.
drwxrwxrwt - hive hive 0 2016-11-17 01:35
/user/hive/warehouse/txntest.db/txntab3/base_0000024
-rw-r--r-- 3 hive hive 451008 2016-11-17 01:35
/user/hive/warehouse/txntest.db/txntab3/base_0000024/bucket_00000
I am going to follow this piece of documentation, so it works:
Configuration Values to Set for Compaction
If the data in your system is not owned by the Hive user (i.e., the user
that the Hive metastore runs as), then Hive will need permission to run as
the user who owns the data in order to perform compactions. If you have
already set up HiveServer2 to impersonate users, then the only additional
work to do is assure that Hive has the right to impersonate users from the
host running the Hive metastore. This is done by adding the hostname to
hadoop.proxyuser.hive.hosts in Hadoop's core-site.xml file. If you have
not already done this, then you will need to configure Hive to act as a
proxy user. This requires you to set up keytabs for the user running the
Hive metastore and add hadoop.proxyuser.hive.hosts and
hadoop.proxyuser.hive.groups to Hadoop's core-site.xml file. See the
Hadoop documentation on secure mode for your version of Hadoop (e.g., for
Hadoop 2.5.1 it is at Hadoop in Secure Mode
<http://hadoop.apache.org/docs/r2.5.1/hadoop-project-dist/hadoop-common/SecureMode.html>
).
On Wed, Nov 16, 2016 at 5:43 PM, Manoj Murumkar <ma...@gmail.com>
wrote:
> Further update:
>
> I see this:
>
> +-----------+----------+------------+--------+--------------
> -------+-----------+----------------+--+
> | dbname | tabname | partname | type | state |
> workerid | starttime |
> +-----------+----------+------------+--------+--------------
> -------+-----------+----------------+--+
> | Database | Table | Partition | Type | State |
> Worker | Start Time |
> | txntest | txntab3 | NULL | MAJOR | ready for cleaning |
> NULL | 1479346924000 |
> +-----------+----------+------------+--------+--------------
> -------+-----------+----------------+--+
>
> However, I don't see cleaner actually run in hive logs (I am looking for a
> string like "compactor.Cleaner").
>
>
> On Wed, Nov 16, 2016 at 5:30 PM, Manoj Murumkar <ma...@gmail.com>
> wrote:
>
>> Quick update:
>>
>> After each compaction, files under base directory (for the buckets) have
>> latest data. However, I am expecting to see all delta files (and
>> directories) gone, as they should be merged in the base directory.
>> Otherwise, we'll start seeing too many small files on HDFS which is a
>> problem. Am I understanding this feature correctly in assuming so?
>>
>> On Wed, Nov 16, 2016 at 5:24 PM, Manoj Murumkar <manoj.murumkar@gmail.com
>> > wrote:
>>
>>> Hi,
>>>
>>> We are trying to implement transaction feature in hive. I created
>>> following table:
>>>
>>> +-----------------------------------------------------------
>>> -----------------------------+--+
>>> | createtab_stmt
>>> |
>>> +-----------------------------------------------------------
>>> -----------------------------+--+
>>> | CREATE TABLE `txntest.txntab3`(
>>> |
>>> | `id` int,
>>> |
>>> | `name` string)
>>> |
>>> | CLUSTERED BY (
>>> |
>>> | id)
>>> |
>>> | INTO 2 BUCKETS
>>> |
>>> | ROW FORMAT SERDE
>>> |
>>> | 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
>>> |
>>> | STORED AS INPUTFORMAT
>>> |
>>> | 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
>>> |
>>> | OUTPUTFORMAT
>>> |
>>> | 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
>>> |
>>> | LOCATION
>>> |
>>> | 'hdfs://or1010051016175.corp.adobe.com:8020/user/hive/wareho
>>> use/txntest.db/txntab3' |
>>> | TBLPROPERTIES (
>>> |
>>> | 'COLUMN_STATS_ACCURATE'='true',
>>> |
>>> | 'numFiles'='22',
>>> |
>>> | 'numRows'='90000',
>>> |
>>> | 'rawDataSize'='0',
>>> |
>>> | 'totalSize'='3564019',
>>> |
>>> | 'transactional'='true',
>>> |
>>> | 'transient_lastDdlTime'='1479329198')
>>> |
>>> +-----------------------------------------------------------
>>> -----------------------------+--+
>>>
>>> I inserted 90000 rows in it in multiple iterations, so it created 22
>>> files (as is visible above). I have run multiple compactions (major and
>>> minor), but nothing seems to happen on HDFS. What am I missing?
>>>
>>> I have following configuration:
>>>
>>> Metastore:
>>>
>>> hive.compactor.initiator.on = true;
>>> hive.compactor.worker.threads = 2;
>>>
>>> Client:
>>>
>>> hive.support.concurrency = true;
>>> hive.exec.dynamic.partition.mode = nonstrict;
>>> hive.txn.manager = org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
>>>
>>> Can someone point me in right direction? Compaction process did run
>>> (verified via SHOW COMPACTIONS) and I also see we have base directory
>>> created on HDFS.
>>> I was expecting all the delta directories gone when major compation runs.
>>>
>>> drwxrwxrwt - admin hive 0 2016-11-16 20:47
>>> /user/hive/warehouse/txntest.db/txntab3/base_0000021
>>> -rw-r--r-- 3 admin hive 227916 2016-11-16 20:47
>>> /user/hive/warehouse/txntest.db/txntab3/base_0000021/bucket_00000
>>> drwxr-xr-x - nex37045 hive 0 2016-11-16 01:33
>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000003_0000003
>>> -rw-r--r-- 3 nex37045 hive 640 2016-11-16 01:33
>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000003_000000
>>> 3/bucket_00000
>>> drwxr-xr-x - nex37045 hive 0 2016-11-16 01:33
>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000004_0000004
>>> -rw-r--r-- 3 nex37045 hive 640 2016-11-16 01:33
>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000004_000000
>>> 4/bucket_00000
>>> drwxr-xr-x - nex37045 hive 0 2016-11-16 01:34
>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000005_0000005
>>> -rw-r--r-- 3 nex37045 hive 640 2016-11-16 01:33
>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000005_000000
>>> 5/bucket_00000
>>> drwxr-xr-x - nex37045 hive 0 2016-11-16 01:34
>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000006_0000006
>>> -rw-r--r-- 3 nex37045 hive 640 2016-11-16 01:34
>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000006_000000
>>> 6/bucket_00000
>>> drwxr-xr-x - nex37045 hive 0 2016-11-16 01:34
>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000007_0000007
>>> -rw-r--r-- 3 nex37045 hive 636 2016-11-16 01:34
>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000007_000000
>>> 7/bucket_00000
>>> drwxr-xr-x - nex37045 hive 0 2016-11-16 01:36
>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000008_0000008
>>> -rw-r--r-- 3 nex37045 hive 640 2016-11-16 01:36
>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000008_000000
>>> 8/bucket_00000
>>> drwxr-xr-x - nex37045 hive 0 2016-11-16 01:36
>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000009_0000009
>>> -rw-r--r-- 3 nex37045 hive 640 2016-11-16 01:36
>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000009_000000
>>> 9/bucket_00000
>>> drwxr-xr-x - nex37045 hive 0 2016-11-16 01:37
>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000010_0000010
>>> -rw-r--r-- 3 nex37045 hive 640 2016-11-16 01:37
>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000010_000001
>>> 0/bucket_00000
>>> drwxr-xr-x - nex37045 hive 0 2016-11-16 01:37
>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000011_0000011
>>> -rw-r--r-- 3 nex37045 hive 644 2016-11-16 01:37
>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000011_000001
>>> 1/bucket_00000
>>> drwxr-xr-x - nex37045 hive 0 2016-11-16 01:37
>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000012_0000012
>>> -rw-r--r-- 3 nex37045 hive 644 2016-11-16 01:37
>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000012_000001
>>> 2/bucket_00000
>>> drwxr-xr-x - nex37045 hive 0 2016-11-16 01:44
>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000013_0000013
>>> -rw-r--r-- 3 nex37045 hive 644 2016-11-16 01:44
>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000013_000001
>>> 3/bucket_00000
>>> drwxr-xr-x - nex37045 hive 0 2016-11-16 01:45
>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000014_0000014
>>> -rw-r--r-- 3 nex37045 hive 644 2016-11-16 01:45
>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000014_000001
>>> 4/bucket_00000
>>> drwxr-xr-x - nex37045 hive 0 2016-11-16 02:02
>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000015_0000015
>>> -rw-r--r-- 3 nex37045 hive 644 2016-11-16 02:02
>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000015_000001
>>> 5/bucket_00000
>>> drwxrwxrwt - admin hive 0 2016-11-16 02:03
>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000015_0000016
>>> -rw-r--r-- 3 admin hive 531 2016-11-16 02:03
>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000015_000001
>>> 6/bucket_00000
>>> drwxr-xr-x - nex37045 hive 0 2016-11-16 02:03
>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000016_0000016
>>> -rw-r--r-- 3 nex37045 hive 644 2016-11-16 02:03
>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000016_000001
>>> 6/bucket_00000
>>> drwxr-xr-x - nex37045 hive 0 2016-11-16 20:37
>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000017_0000017
>>> -rw-r--r-- 3 nex37045 hive 156395 2016-11-16 20:37
>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000017_000001
>>> 7/bucket_00000
>>> drwxrwxrwt - admin hive 0 2016-11-16 20:40
>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000017_0000019
>>> -rw-r--r-- 3 admin hive 2598250 2016-11-16 20:40
>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000017_000001
>>> 9/bucket_00000
>>> drwxr-xr-x - nex37045 hive 0 2016-11-16 20:39
>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000018_0000018
>>> -rw-r--r-- 3 nex37045 hive 4737 2016-11-16 20:39
>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000018_000001
>>> 8/bucket_00000
>>> drwxr-xr-x - nex37045 hive 0 2016-11-16 20:39
>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000019_0000019
>>> -rw-r--r-- 3 nex37045 hive 192658 2016-11-16 20:39
>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000019_000001
>>> 9/bucket_00000
>>> drwxr-xr-x - nex37045 hive 0 2016-11-16 20:45
>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000020_0000020
>>> -rw-r--r-- 3 nex37045 hive 192835 2016-11-16 20:45
>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000020_000002
>>> 0/bucket_00000
>>> drwxr-xr-x - nex37045 hive 0 2016-11-16 20:46
>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000021_0000021
>>> -rw-r--r-- 3 nex37045 hive 201206 2016-11-16 20:46
>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000021_000002
>>> 1/bucket_00000
>>>
>>>
>>>
>>>
>>> Thanks,
>>>
>>> Manoj
>>>
>>>
>>
>
Re: Hive transactions not working
Posted by Manoj Murumkar <ma...@gmail.com>.
Further update:
I see this:
+-----------+----------+------------+--------+---------------------+-----------+----------------+--+
| dbname | tabname | partname | type | state |
workerid | starttime |
+-----------+----------+------------+--------+---------------------+-----------+----------------+--+
| Database | Table | Partition | Type | State |
Worker | Start Time |
| txntest | txntab3 | NULL | MAJOR | ready for cleaning |
NULL | 1479346924000 |
+-----------+----------+------------+--------+---------------------+-----------+----------------+--+
However, I don't see cleaner actually run in hive logs (I am looking for a
string like "compactor.Cleaner").
On Wed, Nov 16, 2016 at 5:30 PM, Manoj Murumkar <ma...@gmail.com>
wrote:
> Quick update:
>
> After each compaction, files under base directory (for the buckets) have
> latest data. However, I am expecting to see all delta files (and
> directories) gone, as they should be merged in the base directory.
> Otherwise, we'll start seeing too many small files on HDFS which is a
> problem. Am I understanding this feature correctly in assuming so?
>
> On Wed, Nov 16, 2016 at 5:24 PM, Manoj Murumkar <ma...@gmail.com>
> wrote:
>
>> Hi,
>>
>> We are trying to implement transaction feature in hive. I created
>> following table:
>>
>> +-----------------------------------------------------------
>> -----------------------------+--+
>> | createtab_stmt
>> |
>> +-----------------------------------------------------------
>> -----------------------------+--+
>> | CREATE TABLE `txntest.txntab3`(
>> |
>> | `id` int,
>> |
>> | `name` string)
>> |
>> | CLUSTERED BY (
>> |
>> | id)
>> |
>> | INTO 2 BUCKETS
>> |
>> | ROW FORMAT SERDE
>> |
>> | 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
>> |
>> | STORED AS INPUTFORMAT
>> |
>> | 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
>> |
>> | OUTPUTFORMAT
>> |
>> | 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
>> |
>> | LOCATION
>> |
>> | 'hdfs://or1010051016175.corp.adobe.com:8020/user/hive/wareho
>> use/txntest.db/txntab3' |
>> | TBLPROPERTIES (
>> |
>> | 'COLUMN_STATS_ACCURATE'='true',
>> |
>> | 'numFiles'='22',
>> |
>> | 'numRows'='90000',
>> |
>> | 'rawDataSize'='0',
>> |
>> | 'totalSize'='3564019',
>> |
>> | 'transactional'='true',
>> |
>> | 'transient_lastDdlTime'='1479329198')
>> |
>> +-----------------------------------------------------------
>> -----------------------------+--+
>>
>> I inserted 90000 rows in it in multiple iterations, so it created 22
>> files (as is visible above). I have run multiple compactions (major and
>> minor), but nothing seems to happen on HDFS. What am I missing?
>>
>> I have following configuration:
>>
>> Metastore:
>>
>> hive.compactor.initiator.on = true;
>> hive.compactor.worker.threads = 2;
>>
>> Client:
>>
>> hive.support.concurrency = true;
>> hive.exec.dynamic.partition.mode = nonstrict;
>> hive.txn.manager = org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
>>
>> Can someone point me in right direction? Compaction process did run
>> (verified via SHOW COMPACTIONS) and I also see we have base directory
>> created on HDFS.
>> I was expecting all the delta directories gone when major compation runs.
>>
>> drwxrwxrwt - admin hive 0 2016-11-16 20:47
>> /user/hive/warehouse/txntest.db/txntab3/base_0000021
>> -rw-r--r-- 3 admin hive 227916 2016-11-16 20:47
>> /user/hive/warehouse/txntest.db/txntab3/base_0000021/bucket_00000
>> drwxr-xr-x - nex37045 hive 0 2016-11-16 01:33
>> /user/hive/warehouse/txntest.db/txntab3/delta_0000003_0000003
>> -rw-r--r-- 3 nex37045 hive 640 2016-11-16 01:33
>> /user/hive/warehouse/txntest.db/txntab3/delta_0000003_000000
>> 3/bucket_00000
>> drwxr-xr-x - nex37045 hive 0 2016-11-16 01:33
>> /user/hive/warehouse/txntest.db/txntab3/delta_0000004_0000004
>> -rw-r--r-- 3 nex37045 hive 640 2016-11-16 01:33
>> /user/hive/warehouse/txntest.db/txntab3/delta_0000004_000000
>> 4/bucket_00000
>> drwxr-xr-x - nex37045 hive 0 2016-11-16 01:34
>> /user/hive/warehouse/txntest.db/txntab3/delta_0000005_0000005
>> -rw-r--r-- 3 nex37045 hive 640 2016-11-16 01:33
>> /user/hive/warehouse/txntest.db/txntab3/delta_0000005_000000
>> 5/bucket_00000
>> drwxr-xr-x - nex37045 hive 0 2016-11-16 01:34
>> /user/hive/warehouse/txntest.db/txntab3/delta_0000006_0000006
>> -rw-r--r-- 3 nex37045 hive 640 2016-11-16 01:34
>> /user/hive/warehouse/txntest.db/txntab3/delta_0000006_000000
>> 6/bucket_00000
>> drwxr-xr-x - nex37045 hive 0 2016-11-16 01:34
>> /user/hive/warehouse/txntest.db/txntab3/delta_0000007_0000007
>> -rw-r--r-- 3 nex37045 hive 636 2016-11-16 01:34
>> /user/hive/warehouse/txntest.db/txntab3/delta_0000007_000000
>> 7/bucket_00000
>> drwxr-xr-x - nex37045 hive 0 2016-11-16 01:36
>> /user/hive/warehouse/txntest.db/txntab3/delta_0000008_0000008
>> -rw-r--r-- 3 nex37045 hive 640 2016-11-16 01:36
>> /user/hive/warehouse/txntest.db/txntab3/delta_0000008_000000
>> 8/bucket_00000
>> drwxr-xr-x - nex37045 hive 0 2016-11-16 01:36
>> /user/hive/warehouse/txntest.db/txntab3/delta_0000009_0000009
>> -rw-r--r-- 3 nex37045 hive 640 2016-11-16 01:36
>> /user/hive/warehouse/txntest.db/txntab3/delta_0000009_000000
>> 9/bucket_00000
>> drwxr-xr-x - nex37045 hive 0 2016-11-16 01:37
>> /user/hive/warehouse/txntest.db/txntab3/delta_0000010_0000010
>> -rw-r--r-- 3 nex37045 hive 640 2016-11-16 01:37
>> /user/hive/warehouse/txntest.db/txntab3/delta_0000010_000001
>> 0/bucket_00000
>> drwxr-xr-x - nex37045 hive 0 2016-11-16 01:37
>> /user/hive/warehouse/txntest.db/txntab3/delta_0000011_0000011
>> -rw-r--r-- 3 nex37045 hive 644 2016-11-16 01:37
>> /user/hive/warehouse/txntest.db/txntab3/delta_0000011_000001
>> 1/bucket_00000
>> drwxr-xr-x - nex37045 hive 0 2016-11-16 01:37
>> /user/hive/warehouse/txntest.db/txntab3/delta_0000012_0000012
>> -rw-r--r-- 3 nex37045 hive 644 2016-11-16 01:37
>> /user/hive/warehouse/txntest.db/txntab3/delta_0000012_000001
>> 2/bucket_00000
>> drwxr-xr-x - nex37045 hive 0 2016-11-16 01:44
>> /user/hive/warehouse/txntest.db/txntab3/delta_0000013_0000013
>> -rw-r--r-- 3 nex37045 hive 644 2016-11-16 01:44
>> /user/hive/warehouse/txntest.db/txntab3/delta_0000013_000001
>> 3/bucket_00000
>> drwxr-xr-x - nex37045 hive 0 2016-11-16 01:45
>> /user/hive/warehouse/txntest.db/txntab3/delta_0000014_0000014
>> -rw-r--r-- 3 nex37045 hive 644 2016-11-16 01:45
>> /user/hive/warehouse/txntest.db/txntab3/delta_0000014_000001
>> 4/bucket_00000
>> drwxr-xr-x - nex37045 hive 0 2016-11-16 02:02
>> /user/hive/warehouse/txntest.db/txntab3/delta_0000015_0000015
>> -rw-r--r-- 3 nex37045 hive 644 2016-11-16 02:02
>> /user/hive/warehouse/txntest.db/txntab3/delta_0000015_000001
>> 5/bucket_00000
>> drwxrwxrwt - admin hive 0 2016-11-16 02:03
>> /user/hive/warehouse/txntest.db/txntab3/delta_0000015_0000016
>> -rw-r--r-- 3 admin hive 531 2016-11-16 02:03
>> /user/hive/warehouse/txntest.db/txntab3/delta_0000015_000001
>> 6/bucket_00000
>> drwxr-xr-x - nex37045 hive 0 2016-11-16 02:03
>> /user/hive/warehouse/txntest.db/txntab3/delta_0000016_0000016
>> -rw-r--r-- 3 nex37045 hive 644 2016-11-16 02:03
>> /user/hive/warehouse/txntest.db/txntab3/delta_0000016_000001
>> 6/bucket_00000
>> drwxr-xr-x - nex37045 hive 0 2016-11-16 20:37
>> /user/hive/warehouse/txntest.db/txntab3/delta_0000017_0000017
>> -rw-r--r-- 3 nex37045 hive 156395 2016-11-16 20:37
>> /user/hive/warehouse/txntest.db/txntab3/delta_0000017_000001
>> 7/bucket_00000
>> drwxrwxrwt - admin hive 0 2016-11-16 20:40
>> /user/hive/warehouse/txntest.db/txntab3/delta_0000017_0000019
>> -rw-r--r-- 3 admin hive 2598250 2016-11-16 20:40
>> /user/hive/warehouse/txntest.db/txntab3/delta_0000017_000001
>> 9/bucket_00000
>> drwxr-xr-x - nex37045 hive 0 2016-11-16 20:39
>> /user/hive/warehouse/txntest.db/txntab3/delta_0000018_0000018
>> -rw-r--r-- 3 nex37045 hive 4737 2016-11-16 20:39
>> /user/hive/warehouse/txntest.db/txntab3/delta_0000018_000001
>> 8/bucket_00000
>> drwxr-xr-x - nex37045 hive 0 2016-11-16 20:39
>> /user/hive/warehouse/txntest.db/txntab3/delta_0000019_0000019
>> -rw-r--r-- 3 nex37045 hive 192658 2016-11-16 20:39
>> /user/hive/warehouse/txntest.db/txntab3/delta_0000019_000001
>> 9/bucket_00000
>> drwxr-xr-x - nex37045 hive 0 2016-11-16 20:45
>> /user/hive/warehouse/txntest.db/txntab3/delta_0000020_0000020
>> -rw-r--r-- 3 nex37045 hive 192835 2016-11-16 20:45
>> /user/hive/warehouse/txntest.db/txntab3/delta_0000020_000002
>> 0/bucket_00000
>> drwxr-xr-x - nex37045 hive 0 2016-11-16 20:46
>> /user/hive/warehouse/txntest.db/txntab3/delta_0000021_0000021
>> -rw-r--r-- 3 nex37045 hive 201206 2016-11-16 20:46
>> /user/hive/warehouse/txntest.db/txntab3/delta_0000021_000002
>> 1/bucket_00000
>>
>>
>>
>>
>> Thanks,
>>
>> Manoj
>>
>>
>
Re: Hive transactions not working
Posted by Manoj Murumkar <ma...@gmail.com>.
Quick update:
After each compaction, files under base directory (for the buckets) have
latest data. However, I am expecting to see all delta files (and
directories) gone, as they should be merged in the base directory.
Otherwise, we'll start seeing too many small files on HDFS which is a
problem. Am I understanding this feature correctly in assuming so?
On Wed, Nov 16, 2016 at 5:24 PM, Manoj Murumkar <ma...@gmail.com>
wrote:
> Hi,
>
> We are trying to implement transaction feature in hive. I created
> following table:
>
> +-----------------------------------------------------------
> -----------------------------+--+
> | createtab_stmt
> |
> +-----------------------------------------------------------
> -----------------------------+--+
> | CREATE TABLE `txntest.txntab3`(
> |
> | `id` int,
> |
> | `name` string)
> |
> | CLUSTERED BY (
> |
> | id)
> |
> | INTO 2 BUCKETS
> |
> | ROW FORMAT SERDE
> |
> | 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> |
> | STORED AS INPUTFORMAT
> |
> | 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> |
> | OUTPUTFORMAT
> |
> | 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> |
> | LOCATION
> |
> | 'hdfs://or1010051016175.corp.adobe.com:8020/user/hive/
> warehouse/txntest.db/txntab3' |
> | TBLPROPERTIES (
> |
> | 'COLUMN_STATS_ACCURATE'='true',
> |
> | 'numFiles'='22',
> |
> | 'numRows'='90000',
> |
> | 'rawDataSize'='0',
> |
> | 'totalSize'='3564019',
> |
> | 'transactional'='true',
> |
> | 'transient_lastDdlTime'='1479329198')
> |
> +-----------------------------------------------------------
> -----------------------------+--+
>
> I inserted 90000 rows in it in multiple iterations, so it created 22 files
> (as is visible above). I have run multiple compactions (major and minor),
> but nothing seems to happen on HDFS. What am I missing?
>
> I have following configuration:
>
> Metastore:
>
> hive.compactor.initiator.on = true;
> hive.compactor.worker.threads = 2;
>
> Client:
>
> hive.support.concurrency = true;
> hive.exec.dynamic.partition.mode = nonstrict;
> hive.txn.manager = org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
>
> Can someone point me in right direction? Compaction process did run
> (verified via SHOW COMPACTIONS) and I also see we have base directory
> created on HDFS.
> I was expecting all the delta directories gone when major compation runs.
>
> drwxrwxrwt - admin hive 0 2016-11-16 20:47
> /user/hive/warehouse/txntest.db/txntab3/base_0000021
> -rw-r--r-- 3 admin hive 227916 2016-11-16 20:47
> /user/hive/warehouse/txntest.db/txntab3/base_0000021/bucket_00000
> drwxr-xr-x - nex37045 hive 0 2016-11-16 01:33
> /user/hive/warehouse/txntest.db/txntab3/delta_0000003_0000003
> -rw-r--r-- 3 nex37045 hive 640 2016-11-16 01:33
> /user/hive/warehouse/txntest.db/txntab3/delta_0000003_0000003/bucket_00000
> drwxr-xr-x - nex37045 hive 0 2016-11-16 01:33
> /user/hive/warehouse/txntest.db/txntab3/delta_0000004_0000004
> -rw-r--r-- 3 nex37045 hive 640 2016-11-16 01:33
> /user/hive/warehouse/txntest.db/txntab3/delta_0000004_0000004/bucket_00000
> drwxr-xr-x - nex37045 hive 0 2016-11-16 01:34
> /user/hive/warehouse/txntest.db/txntab3/delta_0000005_0000005
> -rw-r--r-- 3 nex37045 hive 640 2016-11-16 01:33
> /user/hive/warehouse/txntest.db/txntab3/delta_0000005_0000005/bucket_00000
> drwxr-xr-x - nex37045 hive 0 2016-11-16 01:34
> /user/hive/warehouse/txntest.db/txntab3/delta_0000006_0000006
> -rw-r--r-- 3 nex37045 hive 640 2016-11-16 01:34
> /user/hive/warehouse/txntest.db/txntab3/delta_0000006_0000006/bucket_00000
> drwxr-xr-x - nex37045 hive 0 2016-11-16 01:34
> /user/hive/warehouse/txntest.db/txntab3/delta_0000007_0000007
> -rw-r--r-- 3 nex37045 hive 636 2016-11-16 01:34
> /user/hive/warehouse/txntest.db/txntab3/delta_0000007_0000007/bucket_00000
> drwxr-xr-x - nex37045 hive 0 2016-11-16 01:36
> /user/hive/warehouse/txntest.db/txntab3/delta_0000008_0000008
> -rw-r--r-- 3 nex37045 hive 640 2016-11-16 01:36
> /user/hive/warehouse/txntest.db/txntab3/delta_0000008_0000008/bucket_00000
> drwxr-xr-x - nex37045 hive 0 2016-11-16 01:36
> /user/hive/warehouse/txntest.db/txntab3/delta_0000009_0000009
> -rw-r--r-- 3 nex37045 hive 640 2016-11-16 01:36
> /user/hive/warehouse/txntest.db/txntab3/delta_0000009_0000009/bucket_00000
> drwxr-xr-x - nex37045 hive 0 2016-11-16 01:37
> /user/hive/warehouse/txntest.db/txntab3/delta_0000010_0000010
> -rw-r--r-- 3 nex37045 hive 640 2016-11-16 01:37
> /user/hive/warehouse/txntest.db/txntab3/delta_0000010_0000010/bucket_00000
> drwxr-xr-x - nex37045 hive 0 2016-11-16 01:37
> /user/hive/warehouse/txntest.db/txntab3/delta_0000011_0000011
> -rw-r--r-- 3 nex37045 hive 644 2016-11-16 01:37
> /user/hive/warehouse/txntest.db/txntab3/delta_0000011_0000011/bucket_00000
> drwxr-xr-x - nex37045 hive 0 2016-11-16 01:37
> /user/hive/warehouse/txntest.db/txntab3/delta_0000012_0000012
> -rw-r--r-- 3 nex37045 hive 644 2016-11-16 01:37
> /user/hive/warehouse/txntest.db/txntab3/delta_0000012_0000012/bucket_00000
> drwxr-xr-x - nex37045 hive 0 2016-11-16 01:44
> /user/hive/warehouse/txntest.db/txntab3/delta_0000013_0000013
> -rw-r--r-- 3 nex37045 hive 644 2016-11-16 01:44
> /user/hive/warehouse/txntest.db/txntab3/delta_0000013_0000013/bucket_00000
> drwxr-xr-x - nex37045 hive 0 2016-11-16 01:45
> /user/hive/warehouse/txntest.db/txntab3/delta_0000014_0000014
> -rw-r--r-- 3 nex37045 hive 644 2016-11-16 01:45
> /user/hive/warehouse/txntest.db/txntab3/delta_0000014_0000014/bucket_00000
> drwxr-xr-x - nex37045 hive 0 2016-11-16 02:02
> /user/hive/warehouse/txntest.db/txntab3/delta_0000015_0000015
> -rw-r--r-- 3 nex37045 hive 644 2016-11-16 02:02
> /user/hive/warehouse/txntest.db/txntab3/delta_0000015_0000015/bucket_00000
> drwxrwxrwt - admin hive 0 2016-11-16 02:03
> /user/hive/warehouse/txntest.db/txntab3/delta_0000015_0000016
> -rw-r--r-- 3 admin hive 531 2016-11-16 02:03
> /user/hive/warehouse/txntest.db/txntab3/delta_0000015_0000016/bucket_00000
> drwxr-xr-x - nex37045 hive 0 2016-11-16 02:03
> /user/hive/warehouse/txntest.db/txntab3/delta_0000016_0000016
> -rw-r--r-- 3 nex37045 hive 644 2016-11-16 02:03
> /user/hive/warehouse/txntest.db/txntab3/delta_0000016_0000016/bucket_00000
> drwxr-xr-x - nex37045 hive 0 2016-11-16 20:37
> /user/hive/warehouse/txntest.db/txntab3/delta_0000017_0000017
> -rw-r--r-- 3 nex37045 hive 156395 2016-11-16 20:37
> /user/hive/warehouse/txntest.db/txntab3/delta_0000017_0000017/bucket_00000
> drwxrwxrwt - admin hive 0 2016-11-16 20:40
> /user/hive/warehouse/txntest.db/txntab3/delta_0000017_0000019
> -rw-r--r-- 3 admin hive 2598250 2016-11-16 20:40
> /user/hive/warehouse/txntest.db/txntab3/delta_0000017_0000019/bucket_00000
> drwxr-xr-x - nex37045 hive 0 2016-11-16 20:39
> /user/hive/warehouse/txntest.db/txntab3/delta_0000018_0000018
> -rw-r--r-- 3 nex37045 hive 4737 2016-11-16 20:39
> /user/hive/warehouse/txntest.db/txntab3/delta_0000018_0000018/bucket_00000
> drwxr-xr-x - nex37045 hive 0 2016-11-16 20:39
> /user/hive/warehouse/txntest.db/txntab3/delta_0000019_0000019
> -rw-r--r-- 3 nex37045 hive 192658 2016-11-16 20:39
> /user/hive/warehouse/txntest.db/txntab3/delta_0000019_0000019/bucket_00000
> drwxr-xr-x - nex37045 hive 0 2016-11-16 20:45
> /user/hive/warehouse/txntest.db/txntab3/delta_0000020_0000020
> -rw-r--r-- 3 nex37045 hive 192835 2016-11-16 20:45
> /user/hive/warehouse/txntest.db/txntab3/delta_0000020_0000020/bucket_00000
> drwxr-xr-x - nex37045 hive 0 2016-11-16 20:46
> /user/hive/warehouse/txntest.db/txntab3/delta_0000021_0000021
> -rw-r--r-- 3 nex37045 hive 201206 2016-11-16 20:46
> /user/hive/warehouse/txntest.db/txntab3/delta_0000021_0000021/bucket_00000
>
>
>
>
> Thanks,
>
> Manoj
>
>