You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Arun Allamsetty <ar...@gmail.com> on 2014/08/01 18:24:44 UTC
Re: Best practice for writing to HFileOutputFormat(2) with multiple
Column Families
Hi Jianshi,
Do you mean that you want to sort the row keys? If yes, then you don't have
to worry about it because HBase sorts the row keys on its own but
lexicographically.
Cheers,
Arun
Sent from a mobile device. Please don't mind the typos.
On Jul 30, 2014 9:02 PM, "Jianshi Huang" <ji...@gmail.com> wrote:
> I need to generate from a 2TB dataset and exploded it to 4 Column Families.
>
> The result dataset is likely to be 20TB or more. I'm currently using Spark
> so I sorted the (rk, cf, cq) myself. It's huge and I'm considering how to
> optimize it.
>
> My question is:
> Should I sort and write each column family one by one, or should I put them
> all together then do sort and write?
>
> Does my question make sense?
>
> --
> Jianshi Huang
>
> LinkedIn: jianshi
> Twitter: @jshuang
> Github & Blog: http://huangjs.github.com/
>
Re: Best practice for writing to HFileOutputFormat(2) with multiple
Column Families
Posted by Jianshi Huang <ji...@gmail.com>.
I know HBase will set the TotalOrderPartitioner in MR, but in Spark, I need
to sort the rows myself.
Jianshi
On Sat, Aug 2, 2014 at 12:24 AM, Arun Allamsetty <ar...@gmail.com>
wrote:
> Hi Jianshi,
>
> Do you mean that you want to sort the row keys? If yes, then you don't have
> to worry about it because HBase sorts the row keys on its own but
> lexicographically.
>
> Cheers,
> Arun
>
> Sent from a mobile device. Please don't mind the typos.
> On Jul 30, 2014 9:02 PM, "Jianshi Huang" <ji...@gmail.com> wrote:
>
> > I need to generate from a 2TB dataset and exploded it to 4 Column
> Families.
> >
> > The result dataset is likely to be 20TB or more. I'm currently using
> Spark
> > so I sorted the (rk, cf, cq) myself. It's huge and I'm considering how to
> > optimize it.
> >
> > My question is:
> > Should I sort and write each column family one by one, or should I put
> them
> > all together then do sort and write?
> >
> > Does my question make sense?
> >
> > --
> > Jianshi Huang
> >
> > LinkedIn: jianshi
> > Twitter: @jshuang
> > Github & Blog: http://huangjs.github.com/
> >
>
--
Jianshi Huang
LinkedIn: jianshi
Twitter: @jshuang
Github & Blog: http://huangjs.github.com/