You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@hive.apache.org by Prabhu Joseph <pr...@gmail.com> on 2016/02/17 13:55:51 UTC

Alter Index on Column Query consumes huge memory

Hi All,

   When rebuilding index on a table, where abc is an index on a column
which has 8.5 lakh count for a single column value, the mapreduce job ran
and the reducers timing out, when we monitored the reducers, it is taking
7.3GB RSS and 7.5GB Virt, the reducer memory configured is 7GB. The task is
doing lot of GC and CPU was 2000% and stop the world signal would have
caused the time out of tasks.

    ALTER INDEX abc ON table1 REBUILD;

For 8.5 lakh of BigInt Columns ( 8.5 * 8 bytes = 6MB) , why it needs more
than 7GB. I suspect there is some memory issue with alter index query.


Count      abc
*854959  110335*
395551  80558
340020  275122
208852  308319
192140  139578
179191  49070
165588  346486
133544  199021
130692  270755
103451  275368

Thanks,
Prabhu Joseph