Hive 14 performance and scalability?

[Michael Segel <ms...@hotmail.com>]

Hi, 

While I haven’t upgraded to HDP 2.2, I have to ask if the transaction 
processing introduced in 14 has been tested at scale in terms of both 
users, and data size? 

I am curious as to what happens if you have a long transaction how well  
it copes.


Thx

-Mike

Re: Hive 14 performance and scalability?

[Alan Gates <ga...@hortonworks.com>]

Mike,

I've tested it with inserts, updates, and deletes in the 100k rows per 
partition range, and with multiple but not hundreds of users.  The thing 
to keep in mind is the transactional changes don't affect tables that 
you don't explicitly mark as transactional, so this shouldn't affect 
existing installs or users who aren't interested in the functionality.

Given our design we don't expect long running transactions to be an 
issue, other than it will prevent cleaning up old versions of the data 
used by the long running query, which may result in temporary bloat for 
a table or partition.  Also, keep in mind that currently all DML 
operations are auto-commit so it is harder (though not impossible) to 
get a long running transaction.

Alan.

> Michael Segel <ma...@hotmail.com>
> December 11, 2014 at 3:23
> Hi,
>
> While I haven’t upgraded to HDP 2.2, I have to ask if the transaction
> processing introduced in 14 has been tested at scale in terms of both
> users, and data size?
>
> I am curious as to what happens if you have a long transaction how well
> it copes.
>
>
> Thx
>
> -Mike

-- 
Sent with Postbox <http://www.getpostbox.com>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.