You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Jihong Liu <ji...@gmail.com> on 2014/11/25 19:54:42 UTC
A possible big bug: Delta files created by hive hcatalog streaming
cannot be compacted!
hive hcatalog streaming will also create a file like bucket_n_flush_length
in each delta directory. Where "n" is the bucket number. But
the compactor.CompactorMR think this file also needs to compact. However
this file of course cannot be compacted, so compactor.CompactorMR will not
continue to do the compaction.
Did a test, after removed the bucket_n_flush_length file, then the "alter
table partition compact" finished successfully. If don't delete that file,
nothing will be compacted.
This is probably a very severity bug. Both 0.13 and 0.14 have this issue.
Thanks
Jihong
Re: A possible big bug: Delta files created by hive hcatalog
streaming cannot be compacted!
Posted by Jihong Liu <ji...@gmail.com>.
Alan,
Thank you very much for the quick response. The hive jira: HIVE-8966 (
https://issues.apache.org/jira/browse/HIVE-8966) has been created.
This is really a big issue. We are using hive hcatalog streaming to
populate data into hive. It will be very bad if the data cannot be
compacted. So please fix it as early as possible.
Thank you very much for your help
Jihong
On Tue, Nov 25, 2014 at 11:02 AM, Alan Gates <ga...@hortonworks.com> wrote:
> Can you file a JIRA for this. We can get it fixed and included in the
> 0.14 branch.
>
> Alan.
>
> Jihong Liu <ji...@gmail.com>
> November 25, 2014 at 10:54
> hive hcatalog streaming will also create a file like bucket_n_flush_length
> in each delta directory. Where "n" is the bucket number. But
> the compactor.CompactorMR think this file also needs to compact. However
> this file of course cannot be compacted, so compactor.CompactorMR will not
> continue to do the compaction.
>
> Did a test, after removed the bucket_n_flush_length file, then the "alter
> table partition compact" finished successfully. If don't delete that file,
> nothing will be compacted.
> This is probably a very severity bug. Both 0.13 and 0.14 have this issue.
>
> Thanks
> Jihong
>
>
> --
> Sent with Postbox <http://www.getpostbox.com>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: A possible big bug: Delta files created by hive hcatalog streaming
cannot be compacted!
Posted by Alan Gates <ga...@hortonworks.com>.
Can you file a JIRA for this. We can get it fixed and included in the
0.14 branch.
Alan.
> Jihong Liu <ma...@gmail.com>
> November 25, 2014 at 10:54
> hive hcatalog streaming will also create a file like bucket_n_flush_length
> in each delta directory. Where "n" is the bucket number. But
> the compactor.CompactorMR think this file also needs to compact. However
> this file of course cannot be compacted, so compactor.CompactorMR will not
> continue to do the compaction.
>
> Did a test, after removed the bucket_n_flush_length file, then the "alter
> table partition compact" finished successfully. If don't delete that file,
> nothing will be compacted.
> This is probably a very severity bug. Both 0.13 and 0.14 have this issue.
>
> Thanks
> Jihong
>
--
Sent with Postbox <http://www.getpostbox.com>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.