You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by "yingnan.ma" <yi...@ipinyou.com> on 2012/11/16 10:35:13 UTC
distributed cache
when I use the distributed cache , I found that when the file is more than 100MB or the number of records are more than 10 million ,
the file can not be cache in the memory; and I try to set the io.sort.mb is 200MB ;
it still can not work, Any suggestion would be fine! Thank you !
2012-11-16
Yingnan.Ma
E yingnan.ma@ipinyou.com
MSN: mayingnan_blue@hotmail.com
QQ: 230624226
北京市朝阳区八里庄西里100号东区 住邦2000,1号楼A座2101室,100025
北京・上海・硅谷
http://www.ipinyou.com
Re: distributed cache
Posted by Yanbo Liang <ya...@gmail.com>.
As far as I know, The local.cache.size parameter controls the size of the
DistributedCache. By default, it’s set to 10 GB.
And the parameter io.sort.mb is not used here, it used as each map task has
a circular memory buffer that it writes the output to.
2012/11/16 yingnan.ma <yi...@ipinyou.com>
> **
>
> when I use the distributed cache , I found that when the file is more than 100MB or the number of records are more than 10 million
> ,
>
> the file can not be cache in the memory; and I try to set the io.sort.mb is 200MB ;
> it still can not work, Any suggestion would be fine! Thank you !
>
> 2012-11-16
>
>
>
> Yingnan.Ma
>
> E yingnan.ma@ipinyou.com
>
> MSN: mayingnan_blue@hotmail.com
>
> QQ: 230624226****
>
> 北京市朝阳区八里庄西里100号东区 住邦2000,1号楼A座2101室,100025****
>
> ******************************************************************** ** *
> *************************************************************************
> 北京・上海・硅谷****
>
> http://www.ipinyou.com****
> ------------------------------
>
>
Re: distributed cache
Posted by Yanbo Liang <ya...@gmail.com>.
As far as I know, The local.cache.size parameter controls the size of the
DistributedCache. By default, it’s set to 10 GB.
And the parameter io.sort.mb is not used here, it used as each map task has
a circular memory buffer that it writes the output to.
2012/11/16 yingnan.ma <yi...@ipinyou.com>
> **
>
> when I use the distributed cache , I found that when the file is more than 100MB or the number of records are more than 10 million
> ,
>
> the file can not be cache in the memory; and I try to set the io.sort.mb is 200MB ;
> it still can not work, Any suggestion would be fine! Thank you !
>
> 2012-11-16
>
>
>
> Yingnan.Ma
>
> E yingnan.ma@ipinyou.com
>
> MSN: mayingnan_blue@hotmail.com
>
> QQ: 230624226****
>
> 北京市朝阳区八里庄西里100号东区 住邦2000,1号楼A座2101室,100025****
>
> ******************************************************************** ** *
> *************************************************************************
> 北京・上海・硅谷****
>
> http://www.ipinyou.com****
> ------------------------------
>
>
Re: distributed cache
Posted by Yanbo Liang <ya...@gmail.com>.
As far as I know, The local.cache.size parameter controls the size of the
DistributedCache. By default, it’s set to 10 GB.
And the parameter io.sort.mb is not used here, it used as each map task has
a circular memory buffer that it writes the output to.
2012/11/16 yingnan.ma <yi...@ipinyou.com>
> **
>
> when I use the distributed cache , I found that when the file is more than 100MB or the number of records are more than 10 million
> ,
>
> the file can not be cache in the memory; and I try to set the io.sort.mb is 200MB ;
> it still can not work, Any suggestion would be fine! Thank you !
>
> 2012-11-16
>
>
>
> Yingnan.Ma
>
> E yingnan.ma@ipinyou.com
>
> MSN: mayingnan_blue@hotmail.com
>
> QQ: 230624226****
>
> 北京市朝阳区八里庄西里100号东区 住邦2000,1号楼A座2101室,100025****
>
> ******************************************************************** ** *
> *************************************************************************
> 北京・上海・硅谷****
>
> http://www.ipinyou.com****
> ------------------------------
>
>
Re: distributed cache
Posted by Yanbo Liang <ya...@gmail.com>.
As far as I know, The local.cache.size parameter controls the size of the
DistributedCache. By default, it’s set to 10 GB.
And the parameter io.sort.mb is not used here, it used as each map task has
a circular memory buffer that it writes the output to.
2012/11/16 yingnan.ma <yi...@ipinyou.com>
> **
>
> when I use the distributed cache , I found that when the file is more than 100MB or the number of records are more than 10 million
> ,
>
> the file can not be cache in the memory; and I try to set the io.sort.mb is 200MB ;
> it still can not work, Any suggestion would be fine! Thank you !
>
> 2012-11-16
>
>
>
> Yingnan.Ma
>
> E yingnan.ma@ipinyou.com
>
> MSN: mayingnan_blue@hotmail.com
>
> QQ: 230624226****
>
> 北京市朝阳区八里庄西里100号东区 住邦2000,1号楼A座2101室,100025****
>
> ******************************************************************** ** *
> *************************************************************************
> 北京・上海・硅谷****
>
> http://www.ipinyou.com****
> ------------------------------
>
>