You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Sanjeev Verma <sa...@gmail.com> on 2015/11/13 08:41:30 UTC
hive transaction strange behaviour
I have enable the hive transaction and able to see the delta files created
for some of the partition but i dont not see any base file created yet.it
seems strange to me seeing so many delta files without any base file.
Could somebody let me know when Base file created.
Thanks
Re: hive transaction strange behaviour
Posted by Elliot West <te...@gmail.com>.
It is the compaction process that creates the base files. Check your
configuration to ensure that compaction should be running. I believe the
compactor should run periodically. You can also request a compaction using
the appropriate ALTER TABLE HQL DDL command.
Elliot.
On Friday, 13 November 2015, Sanjeev Verma <sa...@gmail.com>
wrote:
> I have enable the hive transaction and able to see the delta files created
> for some of the partition but i dont not see any base file created yet.it
> seems strange to me seeing so many delta files without any base file.
> Could somebody let me know when Base file created.
>
> Thanks
>
Re: hive transaction strange behaviour
Posted by Eugene Koifman <ek...@hortonworks.com>.
hive.compactor.delta.num.threshold<https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.compactor.delta.num.threshold> controls when compaction is triggered.
If you don't have enough delta files it won't run.
Assuming you have compactions running on other tables/partitions, things must be configured properly.
If you use Alter Table to launch compaction, it should just do it, w/o paying attention to thresholds.
From: Eugene Koifman <ek...@hortonworks.com>>
Reply-To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Date: Wednesday, November 18, 2015 at 10:48 AM
To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Cc: "dev@hive.apache.org<ma...@hive.apache.org>" <de...@hive.apache.org>>
Subject: Re: hive transaction strange behaviour
can you send ls -l on the partition where you expect a base and don't see it?
From: Sanjeev Verma <sa...@gmail.com>>
Reply-To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Date: Tuesday, November 17, 2015 at 10:27 PM
To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Cc: "dev@hive.apache.org<ma...@hive.apache.org>" <de...@hive.apache.org>>
Subject: Re: hive transaction strange behaviour
Any help will be much appreciated.Thanks
On Tue, Nov 17, 2015 at 2:39 PM, Sanjeev Verma <sa...@gmail.com>> wrote:
Thank Elliot, Eugene
I am able to see the Base file created in one of the partition, seems the Compactor kicked in and created it but it has not created base files in rest of the partition where delta files still exists.why compactor has not picked the other partition, when and how these partition will be picked up for compaction.
Thanks
On Sat, Nov 14, 2015 at 11:01 PM, Eugene Koifman <ek...@hortonworks.com>> wrote:
When Compaction process runs, it will create base directory.
https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions#HiveTransactions-Configuration
at a minimum you need hive.compactor.initiator.on<https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.compactor.initiator.on>=true and hive.compactor.worker.threads<https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.compactor.worker.threads>>0
Also, see https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-AlterTable/PartitionCompact on how to trigger compaction manually.
Eugene
From: Sanjeev Verma <sa...@gmail.com>>
Reply-To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Date: Thursday, November 12, 2015 at 11:41 PM
To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>, "dev@hive.apache.org<ma...@hive.apache.org>" <de...@hive.apache.org>>
Subject: hive transaction strange behaviour
I have enable the hive transaction and able to see the delta files created for some of the partition but i dont not see any base file created yet.it<http://yet.it> seems strange to me seeing so many delta files without any base file.
Could somebody let me know when Base file created.
Thanks
Re: hive transaction strange behaviour
Posted by Eugene Koifman <ek...@hortonworks.com>.
hive.compactor.delta.num.threshold<https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.compactor.delta.num.threshold> controls when compaction is triggered.
If you don't have enough delta files it won't run.
Assuming you have compactions running on other tables/partitions, things must be configured properly.
If you use Alter Table to launch compaction, it should just do it, w/o paying attention to thresholds.
From: Eugene Koifman <ek...@hortonworks.com>>
Reply-To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Date: Wednesday, November 18, 2015 at 10:48 AM
To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Cc: "dev@hive.apache.org<ma...@hive.apache.org>" <de...@hive.apache.org>>
Subject: Re: hive transaction strange behaviour
can you send ls -l on the partition where you expect a base and don't see it?
From: Sanjeev Verma <sa...@gmail.com>>
Reply-To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Date: Tuesday, November 17, 2015 at 10:27 PM
To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Cc: "dev@hive.apache.org<ma...@hive.apache.org>" <de...@hive.apache.org>>
Subject: Re: hive transaction strange behaviour
Any help will be much appreciated.Thanks
On Tue, Nov 17, 2015 at 2:39 PM, Sanjeev Verma <sa...@gmail.com>> wrote:
Thank Elliot, Eugene
I am able to see the Base file created in one of the partition, seems the Compactor kicked in and created it but it has not created base files in rest of the partition where delta files still exists.why compactor has not picked the other partition, when and how these partition will be picked up for compaction.
Thanks
On Sat, Nov 14, 2015 at 11:01 PM, Eugene Koifman <ek...@hortonworks.com>> wrote:
When Compaction process runs, it will create base directory.
https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions#HiveTransactions-Configuration
at a minimum you need hive.compactor.initiator.on<https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.compactor.initiator.on>=true and hive.compactor.worker.threads<https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.compactor.worker.threads>>0
Also, see https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-AlterTable/PartitionCompact on how to trigger compaction manually.
Eugene
From: Sanjeev Verma <sa...@gmail.com>>
Reply-To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Date: Thursday, November 12, 2015 at 11:41 PM
To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>, "dev@hive.apache.org<ma...@hive.apache.org>" <de...@hive.apache.org>>
Subject: hive transaction strange behaviour
I have enable the hive transaction and able to see the delta files created for some of the partition but i dont not see any base file created yet.it<http://yet.it> seems strange to me seeing so many delta files without any base file.
Could somebody let me know when Base file created.
Thanks
Re: hive transaction strange behaviour
Posted by Eugene Koifman <ek...@hortonworks.com>.
can you send ls -l on the partition where you expect a base and don't see it?
From: Sanjeev Verma <sa...@gmail.com>>
Reply-To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Date: Tuesday, November 17, 2015 at 10:27 PM
To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Cc: "dev@hive.apache.org<ma...@hive.apache.org>" <de...@hive.apache.org>>
Subject: Re: hive transaction strange behaviour
Any help will be much appreciated.Thanks
On Tue, Nov 17, 2015 at 2:39 PM, Sanjeev Verma <sa...@gmail.com>> wrote:
Thank Elliot, Eugene
I am able to see the Base file created in one of the partition, seems the Compactor kicked in and created it but it has not created base files in rest of the partition where delta files still exists.why compactor has not picked the other partition, when and how these partition will be picked up for compaction.
Thanks
On Sat, Nov 14, 2015 at 11:01 PM, Eugene Koifman <ek...@hortonworks.com>> wrote:
When Compaction process runs, it will create base directory.
https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions#HiveTransactions-Configuration
at a minimum you need hive.compactor.initiator.on<https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.compactor.initiator.on>=true and hive.compactor.worker.threads<https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.compactor.worker.threads>>0
Also, see https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-AlterTable/PartitionCompact on how to trigger compaction manually.
Eugene
From: Sanjeev Verma <sa...@gmail.com>>
Reply-To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Date: Thursday, November 12, 2015 at 11:41 PM
To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>, "dev@hive.apache.org<ma...@hive.apache.org>" <de...@hive.apache.org>>
Subject: hive transaction strange behaviour
I have enable the hive transaction and able to see the delta files created for some of the partition but i dont not see any base file created yet.it<http://yet.it> seems strange to me seeing so many delta files without any base file.
Could somebody let me know when Base file created.
Thanks
Re: hive transaction strange behaviour
Posted by Eugene Koifman <ek...@hortonworks.com>.
can you send ls -l on the partition where you expect a base and don't see it?
From: Sanjeev Verma <sa...@gmail.com>>
Reply-To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Date: Tuesday, November 17, 2015 at 10:27 PM
To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Cc: "dev@hive.apache.org<ma...@hive.apache.org>" <de...@hive.apache.org>>
Subject: Re: hive transaction strange behaviour
Any help will be much appreciated.Thanks
On Tue, Nov 17, 2015 at 2:39 PM, Sanjeev Verma <sa...@gmail.com>> wrote:
Thank Elliot, Eugene
I am able to see the Base file created in one of the partition, seems the Compactor kicked in and created it but it has not created base files in rest of the partition where delta files still exists.why compactor has not picked the other partition, when and how these partition will be picked up for compaction.
Thanks
On Sat, Nov 14, 2015 at 11:01 PM, Eugene Koifman <ek...@hortonworks.com>> wrote:
When Compaction process runs, it will create base directory.
https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions#HiveTransactions-Configuration
at a minimum you need hive.compactor.initiator.on<https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.compactor.initiator.on>=true and hive.compactor.worker.threads<https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.compactor.worker.threads>>0
Also, see https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-AlterTable/PartitionCompact on how to trigger compaction manually.
Eugene
From: Sanjeev Verma <sa...@gmail.com>>
Reply-To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Date: Thursday, November 12, 2015 at 11:41 PM
To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>, "dev@hive.apache.org<ma...@hive.apache.org>" <de...@hive.apache.org>>
Subject: hive transaction strange behaviour
I have enable the hive transaction and able to see the delta files created for some of the partition but i dont not see any base file created yet.it<http://yet.it> seems strange to me seeing so many delta files without any base file.
Could somebody let me know when Base file created.
Thanks
Re: hive transaction strange behaviour
Posted by Sanjeev Verma <sa...@gmail.com>.
Any help will be much appreciated.Thanks
On Tue, Nov 17, 2015 at 2:39 PM, Sanjeev Verma <sa...@gmail.com>
wrote:
> Thank Elliot, Eugene
> I am able to see the Base file created in one of the partition, seems the
> Compactor kicked in and created it but it has not created base files in
> rest of the partition where delta files still exists.why compactor has not
> picked the other partition, when and how these partition will be picked up
> for compaction.
>
> Thanks
>
> On Sat, Nov 14, 2015 at 11:01 PM, Eugene Koifman <ekoifman@hortonworks.com
> > wrote:
>
>> When Compaction process runs, it will create base directory.
>>
>> https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions#HiveTransactions-Configuration
>>
>>
>> at a minimum you need hive.compactor.initiator.on
>> <https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.compactor.initiator.on>=true
>> and hive.compactor.worker.threads
>> <https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.compactor.worker.threads>
>> >0
>>
>> Also, see
>> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-AlterTable/PartitionCompact on
>> how to trigger compaction manually.
>>
>> *Eugene*
>>
>> From: Sanjeev Verma <sa...@gmail.com>
>> Reply-To: "user@hive.apache.org" <us...@hive.apache.org>
>> Date: Thursday, November 12, 2015 at 11:41 PM
>> To: "user@hive.apache.org" <us...@hive.apache.org>, "dev@hive.apache.org"
>> <de...@hive.apache.org>
>> Subject: hive transaction strange behaviour
>>
>> I have enable the hive transaction and able to see the delta files
>> created for some of the partition but i dont not see any base file created
>> yet.it seems strange to me seeing so many delta files without any base
>> file.
>> Could somebody let me know when Base file created.
>>
>> Thanks
>>
>
>
Re: hive transaction strange behaviour
Posted by Sanjeev Verma <sa...@gmail.com>.
Any help will be much appreciated.Thanks
On Tue, Nov 17, 2015 at 2:39 PM, Sanjeev Verma <sa...@gmail.com>
wrote:
> Thank Elliot, Eugene
> I am able to see the Base file created in one of the partition, seems the
> Compactor kicked in and created it but it has not created base files in
> rest of the partition where delta files still exists.why compactor has not
> picked the other partition, when and how these partition will be picked up
> for compaction.
>
> Thanks
>
> On Sat, Nov 14, 2015 at 11:01 PM, Eugene Koifman <ekoifman@hortonworks.com
> > wrote:
>
>> When Compaction process runs, it will create base directory.
>>
>> https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions#HiveTransactions-Configuration
>>
>>
>> at a minimum you need hive.compactor.initiator.on
>> <https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.compactor.initiator.on>=true
>> and hive.compactor.worker.threads
>> <https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.compactor.worker.threads>
>> >0
>>
>> Also, see
>> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-AlterTable/PartitionCompact on
>> how to trigger compaction manually.
>>
>> *Eugene*
>>
>> From: Sanjeev Verma <sa...@gmail.com>
>> Reply-To: "user@hive.apache.org" <us...@hive.apache.org>
>> Date: Thursday, November 12, 2015 at 11:41 PM
>> To: "user@hive.apache.org" <us...@hive.apache.org>, "dev@hive.apache.org"
>> <de...@hive.apache.org>
>> Subject: hive transaction strange behaviour
>>
>> I have enable the hive transaction and able to see the delta files
>> created for some of the partition but i dont not see any base file created
>> yet.it seems strange to me seeing so many delta files without any base
>> file.
>> Could somebody let me know when Base file created.
>>
>> Thanks
>>
>
>
Re: hive transaction strange behaviour
Posted by Sanjeev Verma <sa...@gmail.com>.
Thank Elliot, Eugene
I am able to see the Base file created in one of the partition, seems the
Compactor kicked in and created it but it has not created base files in
rest of the partition where delta files still exists.why compactor has not
picked the other partition, when and how these partition will be picked up
for compaction.
Thanks
On Sat, Nov 14, 2015 at 11:01 PM, Eugene Koifman <ek...@hortonworks.com>
wrote:
> When Compaction process runs, it will create base directory.
>
> https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions#HiveTransactions-Configuration
>
>
> at a minimum you need hive.compactor.initiator.on
> <https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.compactor.initiator.on>=true
> and hive.compactor.worker.threads
> <https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.compactor.worker.threads>
> >0
>
> Also, see
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-AlterTable/PartitionCompact on
> how to trigger compaction manually.
>
> *Eugene*
>
> From: Sanjeev Verma <sa...@gmail.com>
> Reply-To: "user@hive.apache.org" <us...@hive.apache.org>
> Date: Thursday, November 12, 2015 at 11:41 PM
> To: "user@hive.apache.org" <us...@hive.apache.org>, "dev@hive.apache.org" <
> dev@hive.apache.org>
> Subject: hive transaction strange behaviour
>
> I have enable the hive transaction and able to see the delta files created
> for some of the partition but i dont not see any base file created yet.it
> seems strange to me seeing so many delta files without any base file.
> Could somebody let me know when Base file created.
>
> Thanks
>
Re: hive transaction strange behaviour
Posted by Sanjeev Verma <sa...@gmail.com>.
Thank Elliot, Eugene
I am able to see the Base file created in one of the partition, seems the
Compactor kicked in and created it but it has not created base files in
rest of the partition where delta files still exists.why compactor has not
picked the other partition, when and how these partition will be picked up
for compaction.
Thanks
On Sat, Nov 14, 2015 at 11:01 PM, Eugene Koifman <ek...@hortonworks.com>
wrote:
> When Compaction process runs, it will create base directory.
>
> https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions#HiveTransactions-Configuration
>
>
> at a minimum you need hive.compactor.initiator.on
> <https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.compactor.initiator.on>=true
> and hive.compactor.worker.threads
> <https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.compactor.worker.threads>
> >0
>
> Also, see
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-AlterTable/PartitionCompact on
> how to trigger compaction manually.
>
> *Eugene*
>
> From: Sanjeev Verma <sa...@gmail.com>
> Reply-To: "user@hive.apache.org" <us...@hive.apache.org>
> Date: Thursday, November 12, 2015 at 11:41 PM
> To: "user@hive.apache.org" <us...@hive.apache.org>, "dev@hive.apache.org" <
> dev@hive.apache.org>
> Subject: hive transaction strange behaviour
>
> I have enable the hive transaction and able to see the delta files created
> for some of the partition but i dont not see any base file created yet.it
> seems strange to me seeing so many delta files without any base file.
> Could somebody let me know when Base file created.
>
> Thanks
>
Re: hive transaction strange behaviour
Posted by Eugene Koifman <ek...@hortonworks.com>.
When Compaction process runs, it will create base directory.
https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions#HiveTransactions-Configuration
at a minimum you need hive.compactor.initiator.on<https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.compactor.initiator.on>=true and hive.compactor.worker.threads<https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.compactor.worker.threads>>0
Also, see https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-AlterTable/PartitionCompact on how to trigger compaction manually.
Eugene
From: Sanjeev Verma <sa...@gmail.com>>
Reply-To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Date: Thursday, November 12, 2015 at 11:41 PM
To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>, "dev@hive.apache.org<ma...@hive.apache.org>" <de...@hive.apache.org>>
Subject: hive transaction strange behaviour
I have enable the hive transaction and able to see the delta files created for some of the partition but i dont not see any base file created yet.it<http://yet.it> seems strange to me seeing so many delta files without any base file.
Could somebody let me know when Base file created.
Thanks
Re: hive transaction strange behaviour
Posted by Eugene Koifman <ek...@hortonworks.com>.
When Compaction process runs, it will create base directory.
https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions#HiveTransactions-Configuration
at a minimum you need hive.compactor.initiator.on<https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.compactor.initiator.on>=true and hive.compactor.worker.threads<https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.compactor.worker.threads>>0
Also, see https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-AlterTable/PartitionCompact on how to trigger compaction manually.
Eugene
From: Sanjeev Verma <sa...@gmail.com>>
Reply-To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Date: Thursday, November 12, 2015 at 11:41 PM
To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>, "dev@hive.apache.org<ma...@hive.apache.org>" <de...@hive.apache.org>>
Subject: hive transaction strange behaviour
I have enable the hive transaction and able to see the delta files created for some of the partition but i dont not see any base file created yet.it<http://yet.it> seems strange to me seeing so many delta files without any base file.
Could somebody let me know when Base file created.
Thanks