You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Tao Xiao <xi...@gmail.com> on 2013/12/20 08:11:47 UTC
Should we set the property "hbase.local.dir" to a list of directories
on different disks to spread I/O?
Hi,
To spread I/O on multiple disks of the HBase cluster, I set the
property "hbase.local.dir" in hbase-site.xml to a comma-separated list of
directories on different disks as follows:
<property>
<name>hbase.local.dir</name>
<value>/storage/disk1/hbase.local.dir,/storage/disk2/hbase.local.dir,/storage/disk3/hbase.local.dir</value>
</property>
is it right ?
Re: 答复: 答复: Should we set the property "hbase.local.dir" to a list of directories on different disks to spread I/O?
Posted by Tao Xiao <xi...@gmail.com>.
To all:
Thank you all.
Now I understand how to set "hbase.local.dir" and the reason
2013/12/20 Jean-Marc Spaggiari <je...@spaggiari.org>
> Barath's respons is a good summary.
>
> There almost nothing going into hbase.local.dir. It's like in your local
> linux box. You don't specify more than one tmp directory. It's the same
> thing here. There will not be really any benefit of adding that.
>
>
> 2013/12/20 Bharath Vissapragada <bh...@cloudera.com>
>
> > Hbase uses HDFS for data storage and region servers just use HDFS API to
> > write files. So you should be looking at balancing disk IO at datanode
> > level (dfs.datanode.data.dir as mentioned in other response). The path
> you
> > mentioned (hbase.local.dir) is just used for storing temporary files or
> > CP/dynamic jars etc (not data) by region server and doesn't take much
> space
> > / cause IO.
> >
> >
> > On Fri, Dec 20, 2013 at 2:49 PM, 谢良 <xi...@xiaomi.com> wrote:
> >
> > > maybe you have a misunderstanding about "hbase.local.dir", please refer
> > to
> > > HBASE-6824 for why it was introduced.
> > >
> > > Thanks,
> > > ________________________________________
> > > 发件人: Tao Xiao [xiaotao.cs.nju@gmail.com]
> > > 发送时间: 2013年12月20日 16:30
> > > 收件人: user
> > > 主题: Re: 答复: Should we set the property "hbase.local.dir" to a list of
> > > directories on different disks to spread I/O?
> > >
> > > Since "hbase.local.dir" specifes local directory, why doesn't it
> specify
> > > multiple directories on different disks to spread I/O, just as what
> > > "dfs.datanode.data.dir" does ?
> > >
> > >
> > > 2013/12/20 谢良 <xi...@xiaomi.com>
> > >
> > > > Please take a look at "dfs.datanode.data.dir" in hdfs-site.xml.
> > > > P.S. "hbase.local.dir" supports one dir only, don't specify it with a
> > > > list:)
> > > >
> > > > Thanks,
> > > > ________________________________________
> > > > 发件人: Tao Xiao [xiaotao.cs.nju@gmail.com]
> > > > 发送时间: 2013年12月20日 15:11
> > > > 收件人: user
> > > > 主题: Should we set the property "hbase.local.dir" to a list of
> > directories
> > > > on different disks to spread I/O?
> > > >
> > > > Hi,
> > > >
> > > > To spread I/O on multiple disks of the HBase cluster, I set the
> > > > property "hbase.local.dir" in hbase-site.xml to a comma-separated
> list
> > > of
> > > > directories on different disks as follows:
> > > >
> > > > <property>
> > > > <name>hbase.local.dir</name>
> > > >
> > > >
> > > >
> > >
> >
> <value>/storage/disk1/hbase.local.dir,/storage/disk2/hbase.local.dir,/storage/disk3/hbase.local.dir</value>
> > > > </property>
> > > >
> > > > is it right ?
> > > >
> > >
> >
> >
> >
> > --
> > Bharath Vissapragada
> > <http://www.cloudera.com>
> >
>
Re: 答复: 答复: Should we set the property "hbase.local.dir" to a list of directories on different disks to spread I/O?
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Barath's respons is a good summary.
There almost nothing going into hbase.local.dir. It's like in your local
linux box. You don't specify more than one tmp directory. It's the same
thing here. There will not be really any benefit of adding that.
2013/12/20 Bharath Vissapragada <bh...@cloudera.com>
> Hbase uses HDFS for data storage and region servers just use HDFS API to
> write files. So you should be looking at balancing disk IO at datanode
> level (dfs.datanode.data.dir as mentioned in other response). The path you
> mentioned (hbase.local.dir) is just used for storing temporary files or
> CP/dynamic jars etc (not data) by region server and doesn't take much space
> / cause IO.
>
>
> On Fri, Dec 20, 2013 at 2:49 PM, 谢良 <xi...@xiaomi.com> wrote:
>
> > maybe you have a misunderstanding about "hbase.local.dir", please refer
> to
> > HBASE-6824 for why it was introduced.
> >
> > Thanks,
> > ________________________________________
> > 发件人: Tao Xiao [xiaotao.cs.nju@gmail.com]
> > 发送时间: 2013年12月20日 16:30
> > 收件人: user
> > 主题: Re: 答复: Should we set the property "hbase.local.dir" to a list of
> > directories on different disks to spread I/O?
> >
> > Since "hbase.local.dir" specifes local directory, why doesn't it specify
> > multiple directories on different disks to spread I/O, just as what
> > "dfs.datanode.data.dir" does ?
> >
> >
> > 2013/12/20 谢良 <xi...@xiaomi.com>
> >
> > > Please take a look at "dfs.datanode.data.dir" in hdfs-site.xml.
> > > P.S. "hbase.local.dir" supports one dir only, don't specify it with a
> > > list:)
> > >
> > > Thanks,
> > > ________________________________________
> > > 发件人: Tao Xiao [xiaotao.cs.nju@gmail.com]
> > > 发送时间: 2013年12月20日 15:11
> > > 收件人: user
> > > 主题: Should we set the property "hbase.local.dir" to a list of
> directories
> > > on different disks to spread I/O?
> > >
> > > Hi,
> > >
> > > To spread I/O on multiple disks of the HBase cluster, I set the
> > > property "hbase.local.dir" in hbase-site.xml to a comma-separated list
> > of
> > > directories on different disks as follows:
> > >
> > > <property>
> > > <name>hbase.local.dir</name>
> > >
> > >
> > >
> >
> <value>/storage/disk1/hbase.local.dir,/storage/disk2/hbase.local.dir,/storage/disk3/hbase.local.dir</value>
> > > </property>
> > >
> > > is it right ?
> > >
> >
>
>
>
> --
> Bharath Vissapragada
> <http://www.cloudera.com>
>
Re: 答复: 答复: Should we set the property "hbase.local.dir" to a list of directories on different disks to spread I/O?
Posted by Bharath Vissapragada <bh...@cloudera.com>.
Hbase uses HDFS for data storage and region servers just use HDFS API to
write files. So you should be looking at balancing disk IO at datanode
level (dfs.datanode.data.dir as mentioned in other response). The path you
mentioned (hbase.local.dir) is just used for storing temporary files or
CP/dynamic jars etc (not data) by region server and doesn't take much space
/ cause IO.
On Fri, Dec 20, 2013 at 2:49 PM, 谢良 <xi...@xiaomi.com> wrote:
> maybe you have a misunderstanding about "hbase.local.dir", please refer to
> HBASE-6824 for why it was introduced.
>
> Thanks,
> ________________________________________
> 发件人: Tao Xiao [xiaotao.cs.nju@gmail.com]
> 发送时间: 2013年12月20日 16:30
> 收件人: user
> 主题: Re: 答复: Should we set the property "hbase.local.dir" to a list of
> directories on different disks to spread I/O?
>
> Since "hbase.local.dir" specifes local directory, why doesn't it specify
> multiple directories on different disks to spread I/O, just as what
> "dfs.datanode.data.dir" does ?
>
>
> 2013/12/20 谢良 <xi...@xiaomi.com>
>
> > Please take a look at "dfs.datanode.data.dir" in hdfs-site.xml.
> > P.S. "hbase.local.dir" supports one dir only, don't specify it with a
> > list:)
> >
> > Thanks,
> > ________________________________________
> > 发件人: Tao Xiao [xiaotao.cs.nju@gmail.com]
> > 发送时间: 2013年12月20日 15:11
> > 收件人: user
> > 主题: Should we set the property "hbase.local.dir" to a list of directories
> > on different disks to spread I/O?
> >
> > Hi,
> >
> > To spread I/O on multiple disks of the HBase cluster, I set the
> > property "hbase.local.dir" in hbase-site.xml to a comma-separated list
> of
> > directories on different disks as follows:
> >
> > <property>
> > <name>hbase.local.dir</name>
> >
> >
> >
> <value>/storage/disk1/hbase.local.dir,/storage/disk2/hbase.local.dir,/storage/disk3/hbase.local.dir</value>
> > </property>
> >
> > is it right ?
> >
>
--
Bharath Vissapragada
<http://www.cloudera.com>
答复: 答复: Should we set the property "hbase.local.dir" to a list of directories on different disks to spread I/O?
Posted by 谢良 <xi...@xiaomi.com>.
maybe you have a misunderstanding about "hbase.local.dir", please refer to HBASE-6824 for why it was introduced.
Thanks,
________________________________________
发件人: Tao Xiao [xiaotao.cs.nju@gmail.com]
发送时间: 2013年12月20日 16:30
收件人: user
主题: Re: 答复: Should we set the property "hbase.local.dir" to a list of directories on different disks to spread I/O?
Since "hbase.local.dir" specifes local directory, why doesn't it specify
multiple directories on different disks to spread I/O, just as what
"dfs.datanode.data.dir" does ?
2013/12/20 谢良 <xi...@xiaomi.com>
> Please take a look at "dfs.datanode.data.dir" in hdfs-site.xml.
> P.S. "hbase.local.dir" supports one dir only, don't specify it with a
> list:)
>
> Thanks,
> ________________________________________
> 发件人: Tao Xiao [xiaotao.cs.nju@gmail.com]
> 发送时间: 2013年12月20日 15:11
> 收件人: user
> 主题: Should we set the property "hbase.local.dir" to a list of directories
> on different disks to spread I/O?
>
> Hi,
>
> To spread I/O on multiple disks of the HBase cluster, I set the
> property "hbase.local.dir" in hbase-site.xml to a comma-separated list of
> directories on different disks as follows:
>
> <property>
> <name>hbase.local.dir</name>
>
>
> <value>/storage/disk1/hbase.local.dir,/storage/disk2/hbase.local.dir,/storage/disk3/hbase.local.dir</value>
> </property>
>
> is it right ?
>
Re: 答复: Should we set the property "hbase.local.dir" to a list of directories on different disks to spread I/O?
Posted by Tao Xiao <xi...@gmail.com>.
Since "hbase.local.dir" specifes local directory, why doesn't it specify
multiple directories on different disks to spread I/O, just as what
"dfs.datanode.data.dir" does ?
2013/12/20 谢良 <xi...@xiaomi.com>
> Please take a look at "dfs.datanode.data.dir" in hdfs-site.xml.
> P.S. "hbase.local.dir" supports one dir only, don't specify it with a
> list:)
>
> Thanks,
> ________________________________________
> 发件人: Tao Xiao [xiaotao.cs.nju@gmail.com]
> 发送时间: 2013年12月20日 15:11
> 收件人: user
> 主题: Should we set the property "hbase.local.dir" to a list of directories
> on different disks to spread I/O?
>
> Hi,
>
> To spread I/O on multiple disks of the HBase cluster, I set the
> property "hbase.local.dir" in hbase-site.xml to a comma-separated list of
> directories on different disks as follows:
>
> <property>
> <name>hbase.local.dir</name>
>
>
> <value>/storage/disk1/hbase.local.dir,/storage/disk2/hbase.local.dir,/storage/disk3/hbase.local.dir</value>
> </property>
>
> is it right ?
>
答复: Should we set the property "hbase.local.dir" to a list of directories on different disks to spread I/O?
Posted by 谢良 <xi...@xiaomi.com>.
Please take a look at "dfs.datanode.data.dir" in hdfs-site.xml.
P.S. "hbase.local.dir" supports one dir only, don't specify it with a list:)
Thanks,
________________________________________
发件人: Tao Xiao [xiaotao.cs.nju@gmail.com]
发送时间: 2013年12月20日 15:11
收件人: user
主题: Should we set the property "hbase.local.dir" to a list of directories on different disks to spread I/O?
Hi,
To spread I/O on multiple disks of the HBase cluster, I set the
property "hbase.local.dir" in hbase-site.xml to a comma-separated list of
directories on different disks as follows:
<property>
<name>hbase.local.dir</name>
<value>/storage/disk1/hbase.local.dir,/storage/disk2/hbase.local.dir,/storage/disk3/hbase.local.dir</value>
</property>
is it right ?