Row group alignment?

[Ryan Blue <bl...@cloudera.com>]

Hi everyone,

I have a question for anyone with large datasets in HDFS about row group 
and HDFS block alignment: how well are row groups and HFDS blocks 
aligned in practice?

Say your row group size is equal to HDFS block size, then how many 
blocks does it take, on average, before a row group is significantly 
split between two blocks? Or put differently, how much shorter are row 
groups than the planned row group size, on average?

I'm trying to find out whether it would be a significant benefit to use 
something like variable-length HDFS blocks, added in HDFS-3689 [1], to 
keep the two aligned.

Thanks!

rb

[1]: https://issues.apache.org/jira/browse/HDFS-3689


-- 
Ryan Blue
Software Engineer
Cloudera, Inc.