You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Jean-Marc Spaggiari <je...@spaggiari.org> on 2013/11/26 04:58:58 UTC
Re: HFile block size
>From the code and the JIRAs:
hbase.hregion.max.filesize is used to configure the size of a region (which
can contain more than one HFile)
hbase.mapreduce.hfileoutputformat.blocksize come from HBase 8949 (While
writing hfiles from HFileOutputFormat forcing blocksize from table
schema(HColumnDescriptor).
Even if we configure hbase.mapreduce.hfileoutputformat.blocksize during
bulkload/import it
will be overridden with actual block size from table schema. )
hfile.min.blocksize.size is the old
hbase.mapreduce.hfileoutputformat.blocksize (See HBase-3864)
2013/11/25 Job Thomas <jo...@suntecgroup.com>
> Hi all,
>
> Out of these property , which one is used to set HFile block size in hbae
> 0.94.12
>
> hbase.hregion.max.filesize=16384
>
> hfile.min.blocksize.size=16384
>
> hbase.mapreduce.hfileoutputformat.blocksize=16384
>
> Best Regards,
> Job M Thomas
>
RE: HFile block size [Resolved]
Posted by Job Thomas <jo...@suntecgroup.com>.
Hi Jean & Ted,
Thank You,
I have created table manually in hbase with blocksize='8192' ( create 'TESTBLOCK', {NAME => 'cf', BLOCKSIZE => '8192'} ) , and then created table with same name via Phoenix.
Best Regards,
Job M Thomas
________________________________
From: Job Thomas
Sent: Tue 11/26/2013 10:06 AM
To: user@hbase.apache.org; user
Subject: RE: HFile block size
Hi Jean ,
Thanks You for the support.
we can create table like this , create 'xyz', {NAME => 'cf', BLOCKSIZE => '8192'} inorder to set the block size.
But I am using phoenix to create and query the table. I want to globaly declare this property to let all table crated should take this property.
My priamry aim is to decrease the random read latency by reducing block size.
how can I make that?
Best Regards,
Job M Thomas
________________________________
From: Jean-Marc Spaggiari [mailto:jean-marc@spaggiari.org]
Sent: Tue 11/26/2013 9:28 AM
To: user
Subject: Re: HFile block size
From the code and the JIRAs:
hbase.hregion.max.filesize is used to configure the size of a region (which
can contain more than one HFile)
hbase.mapreduce.hfileoutputformat.blocksize come from HBase 8949 (While
writing hfiles from HFileOutputFormat forcing blocksize from table
schema(HColumnDescriptor).
Even if we configure hbase.mapreduce.hfileoutputformat.blocksize during
bulkload/import it
will be overridden with actual block size from table schema. )
hfile.min.blocksize.size is the old
hbase.mapreduce.hfileoutputformat.blocksize (See HBase-3864)
2013/11/25 Job Thomas <jo...@suntecgroup.com>
> Hi all,
>
> Out of these property , which one is used to set HFile block size in hbae
> 0.94.12
>
> hbase.hregion.max.filesize=16384
>
> hfile.min.blocksize.size=16384
>
> hbase.mapreduce.hfileoutputformat.blocksize=16384
>
> Best Regards,
> Job M Thomas
>
RE: HFile block size
Posted by Job Thomas <jo...@suntecgroup.com>.
Hi James,
It is working , Thank you for your help . The query latency of a 10 million table has been reduced to 15 millisecond from 28 millisecond by reducing block size of the table.
With Thanks,
Job M Thomas
________________________________
From: James Taylor [mailto:jtaylor@salesforce.com]
Sent: Tue 11/26/2013 10:42 AM
To: user@hbase.apache.org
Subject: Re: HFile block size
FYI, you can define BLOCKSIZE in your hbase-sites.xml, just like with HBase
to make it global.
Thanks,
James
On Mon, Nov 25, 2013 at 9:08 PM, Azuryy Yu <az...@gmail.com> wrote:
> This is no way to declare global property in Phoneix, you have to
> declare BLOCKSIZE
> in each 'create' SQL.
>
> such as:
> "CREATE TABLE IF NOT EXISTS STOCK_SYMBOL(id int, name string)
> BLOOMFILTER='ROW', VERSIONS='1', BLOCKSIZE => '8192' "
>
>
> On Tue, Nov 26, 2013 at 12:36 PM, Job Thomas <jo...@suntecgroup.com> wrote:
>
> >
> > Hi Jean ,
> >
> > Thanks You for the support.
> >
> > we can create table like this , create 'xyz', {NAME => 'cf', BLOCKSIZE
> =>
> > '8192'} inorder to set the block size.
> >
> > But I am using phoenix to create and query the table. I want to globaly
> > declare this property to let all table crated should take this property.
> >
> > My priamry aim is to decrease the random read latency by reducing block
> > size.
> >
> > how can I make that?
> >
> > Best Regards,
> > Job M Thomas
> >
> > ________________________________
> >
> > From: Jean-Marc Spaggiari [mailto:jean-marc@spaggiari.org]
> > Sent: Tue 11/26/2013 9:28 AM
> > To: user
> > Subject: Re: HFile block size
> >
> >
> >
> > From the code and the JIRAs:
> >
> > hbase.hregion.max.filesize is used to configure the size of a region
> (which
> > can contain more than one HFile)
> >
> > hbase.mapreduce.hfileoutputformat.blocksize come from HBase 8949 (While
> > writing hfiles from HFileOutputFormat forcing blocksize from table
> > schema(HColumnDescriptor).
> > Even if we configure hbase.mapreduce.hfileoutputformat.blocksize during
> > bulkload/import it
> > will be overridden with actual block size from table schema. )
> >
> > hfile.min.blocksize.size is the old
> > hbase.mapreduce.hfileoutputformat.blocksize (See HBase-3864)
> >
> >
> > 2013/11/25 Job Thomas <jo...@suntecgroup.com>
> >
> > > Hi all,
> > >
> > > Out of these property , which one is used to set HFile block size in
> > hbae
> > > 0.94.12
> > >
> > > hbase.hregion.max.filesize=16384
> > >
> > > hfile.min.blocksize.size=16384
> > >
> > > hbase.mapreduce.hfileoutputformat.blocksize=16384
> > >
> > > Best Regards,
> > > Job M Thomas
> > >
> >
> >
> >
>
Re: HFile block size
Posted by James Taylor <jt...@salesforce.com>.
FYI, you can define BLOCKSIZE in your hbase-sites.xml, just like with HBase
to make it global.
Thanks,
James
On Mon, Nov 25, 2013 at 9:08 PM, Azuryy Yu <az...@gmail.com> wrote:
> This is no way to declare global property in Phoneix, you have to
> declare BLOCKSIZE
> in each 'create' SQL.
>
> such as:
> "CREATE TABLE IF NOT EXISTS STOCK_SYMBOL(id int, name string)
> BLOOMFILTER='ROW', VERSIONS='1', BLOCKSIZE => '8192' "
>
>
> On Tue, Nov 26, 2013 at 12:36 PM, Job Thomas <jo...@suntecgroup.com> wrote:
>
> >
> > Hi Jean ,
> >
> > Thanks You for the support.
> >
> > we can create table like this , create 'xyz', {NAME => 'cf', BLOCKSIZE
> =>
> > '8192'} inorder to set the block size.
> >
> > But I am using phoenix to create and query the table. I want to globaly
> > declare this property to let all table crated should take this property.
> >
> > My priamry aim is to decrease the random read latency by reducing block
> > size.
> >
> > how can I make that?
> >
> > Best Regards,
> > Job M Thomas
> >
> > ________________________________
> >
> > From: Jean-Marc Spaggiari [mailto:jean-marc@spaggiari.org]
> > Sent: Tue 11/26/2013 9:28 AM
> > To: user
> > Subject: Re: HFile block size
> >
> >
> >
> > From the code and the JIRAs:
> >
> > hbase.hregion.max.filesize is used to configure the size of a region
> (which
> > can contain more than one HFile)
> >
> > hbase.mapreduce.hfileoutputformat.blocksize come from HBase 8949 (While
> > writing hfiles from HFileOutputFormat forcing blocksize from table
> > schema(HColumnDescriptor).
> > Even if we configure hbase.mapreduce.hfileoutputformat.blocksize during
> > bulkload/import it
> > will be overridden with actual block size from table schema. )
> >
> > hfile.min.blocksize.size is the old
> > hbase.mapreduce.hfileoutputformat.blocksize (See HBase-3864)
> >
> >
> > 2013/11/25 Job Thomas <jo...@suntecgroup.com>
> >
> > > Hi all,
> > >
> > > Out of these property , which one is used to set HFile block size in
> > hbae
> > > 0.94.12
> > >
> > > hbase.hregion.max.filesize=16384
> > >
> > > hfile.min.blocksize.size=16384
> > >
> > > hbase.mapreduce.hfileoutputformat.blocksize=16384
> > >
> > > Best Regards,
> > > Job M Thomas
> > >
> >
> >
> >
>
Re: HFile block size
Posted by Azuryy Yu <az...@gmail.com>.
This is no way to declare global property in Phoneix, you have to
declare BLOCKSIZE
in each 'create' SQL.
such as:
"CREATE TABLE IF NOT EXISTS STOCK_SYMBOL(id int, name string)
BLOOMFILTER='ROW', VERSIONS='1', BLOCKSIZE => '8192' "
On Tue, Nov 26, 2013 at 12:36 PM, Job Thomas <jo...@suntecgroup.com> wrote:
>
> Hi Jean ,
>
> Thanks You for the support.
>
> we can create table like this , create 'xyz', {NAME => 'cf', BLOCKSIZE =>
> '8192'} inorder to set the block size.
>
> But I am using phoenix to create and query the table. I want to globaly
> declare this property to let all table crated should take this property.
>
> My priamry aim is to decrease the random read latency by reducing block
> size.
>
> how can I make that?
>
> Best Regards,
> Job M Thomas
>
> ________________________________
>
> From: Jean-Marc Spaggiari [mailto:jean-marc@spaggiari.org]
> Sent: Tue 11/26/2013 9:28 AM
> To: user
> Subject: Re: HFile block size
>
>
>
> From the code and the JIRAs:
>
> hbase.hregion.max.filesize is used to configure the size of a region (which
> can contain more than one HFile)
>
> hbase.mapreduce.hfileoutputformat.blocksize come from HBase 8949 (While
> writing hfiles from HFileOutputFormat forcing blocksize from table
> schema(HColumnDescriptor).
> Even if we configure hbase.mapreduce.hfileoutputformat.blocksize during
> bulkload/import it
> will be overridden with actual block size from table schema. )
>
> hfile.min.blocksize.size is the old
> hbase.mapreduce.hfileoutputformat.blocksize (See HBase-3864)
>
>
> 2013/11/25 Job Thomas <jo...@suntecgroup.com>
>
> > Hi all,
> >
> > Out of these property , which one is used to set HFile block size in
> hbae
> > 0.94.12
> >
> > hbase.hregion.max.filesize=16384
> >
> > hfile.min.blocksize.size=16384
> >
> > hbase.mapreduce.hfileoutputformat.blocksize=16384
> >
> > Best Regards,
> > Job M Thomas
> >
>
>
>
RE: HFile block size
Posted by Job Thomas <jo...@suntecgroup.com>.
Hi Jean ,
Thanks You for the support.
we can create table like this , create 'xyz', {NAME => 'cf', BLOCKSIZE => '8192'} inorder to set the block size.
But I am using phoenix to create and query the table. I want to globaly declare this property to let all table crated should take this property.
My priamry aim is to decrease the random read latency by reducing block size.
how can I make that?
Best Regards,
Job M Thomas
________________________________
From: Jean-Marc Spaggiari [mailto:jean-marc@spaggiari.org]
Sent: Tue 11/26/2013 9:28 AM
To: user
Subject: Re: HFile block size
From the code and the JIRAs:
hbase.hregion.max.filesize is used to configure the size of a region (which
can contain more than one HFile)
hbase.mapreduce.hfileoutputformat.blocksize come from HBase 8949 (While
writing hfiles from HFileOutputFormat forcing blocksize from table
schema(HColumnDescriptor).
Even if we configure hbase.mapreduce.hfileoutputformat.blocksize during
bulkload/import it
will be overridden with actual block size from table schema. )
hfile.min.blocksize.size is the old
hbase.mapreduce.hfileoutputformat.blocksize (See HBase-3864)
2013/11/25 Job Thomas <jo...@suntecgroup.com>
> Hi all,
>
> Out of these property , which one is used to set HFile block size in hbae
> 0.94.12
>
> hbase.hregion.max.filesize=16384
>
> hfile.min.blocksize.size=16384
>
> hbase.mapreduce.hfileoutputformat.blocksize=16384
>
> Best Regards,
> Job M Thomas
>
Re: HFile block size
Posted by Ted Yu <yu...@gmail.com>.
Job:
Please take a look at http://hbase.apache.org/book.html#schema.cf.blocksize
On Tue, Nov 26, 2013 at 11:58 AM, Jean-Marc Spaggiari <
jean-marc@spaggiari.org> wrote:
> From the code and the JIRAs:
>
> hbase.hregion.max.filesize is used to configure the size of a region (which
> can contain more than one HFile)
>
> hbase.mapreduce.hfileoutputformat.blocksize come from HBase 8949 (While
> writing hfiles from HFileOutputFormat forcing blocksize from table
> schema(HColumnDescriptor).
> Even if we configure hbase.mapreduce.hfileoutputformat.blocksize during
> bulkload/import it
> will be overridden with actual block size from table schema. )
>
> hfile.min.blocksize.size is the old
> hbase.mapreduce.hfileoutputformat.blocksize (See HBase-3864)
>
>
> 2013/11/25 Job Thomas <jo...@suntecgroup.com>
>
> > Hi all,
> >
> > Out of these property , which one is used to set HFile block size in
> hbae
> > 0.94.12
> >
> > hbase.hregion.max.filesize=16384
> >
> > hfile.min.blocksize.size=16384
> >
> > hbase.mapreduce.hfileoutputformat.blocksize=16384
> >
> > Best Regards,
> > Job M Thomas
> >
>