You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Kurt Moesky <ku...@gmail.com> on 2014/01/03 17:37:50 UTC
Block size
I see the default block size for HDFS is 64 MB, is this a value that can be
changed easily?
Re: Block size
Posted by Harsh J <ha...@cloudera.com>.
XG,
The newer default is 128 MB [HDFS-4053]. The minimum, however, can be
as low as io.bytes.per.checksum (default: 512 bytes) if the user so
wishes it. To administratively set a limit to prevent low values from
being used, see the config introduced via HDFS-4305.
On Sat, Jan 4, 2014 at 11:38 AM, Zhao, Xiaoguang
<Xi...@honeywell.com> wrote:
> As I am new to hdfs, I was told that the minimize block size is 64M, is it
> correct?
>
> XG
>
> 在 2014年1月4日，3:12，"German Florez-Larrahondo" <ge...@samsung.com> 写道：
>
> Also note that the block size in recent releases is actually called
> “dfs.blocksize” as opposed to “dfs.block.size”, and that you can set it per
> job as well. In that scenario, just pass it as an argument to your job (e.g.
> Hadoop bla –D dfs.blocksize= 134217728)
>
>
>
> Regards
>
>
>
> From: David Sinclair [mailto:dsinclair@chariotsolutions.com]
> Sent: Friday, January 03, 2014 10:47 AM
> To: user@hadoop.apache.org
> Subject: Re: Block size
>
>
>
> Change the dfs.block.size in hdfs-site.xml to be the value you would like if
> you want to have all new files have a different block size.
>
>
>
> On Fri, Jan 3, 2014 at 11:37 AM, Kurt Moesky <ku...@gmail.com> wrote:
>
> I see the default block size for HDFS is 64 MB, is this a value that can be
> changed easily?
>
>
--
Harsh J
Re: Block size
Posted by Harsh J <ha...@cloudera.com>.
XG,
The newer default is 128 MB [HDFS-4053]. The minimum, however, can be
as low as io.bytes.per.checksum (default: 512 bytes) if the user so
wishes it. To administratively set a limit to prevent low values from
being used, see the config introduced via HDFS-4305.
On Sat, Jan 4, 2014 at 11:38 AM, Zhao, Xiaoguang
<Xi...@honeywell.com> wrote:
> As I am new to hdfs, I was told that the minimize block size is 64M, is it
> correct?
>
> XG
>
> 在 2014年1月4日，3:12，"German Florez-Larrahondo" <ge...@samsung.com> 写道：
>
> Also note that the block size in recent releases is actually called
> “dfs.blocksize” as opposed to “dfs.block.size”, and that you can set it per
> job as well. In that scenario, just pass it as an argument to your job (e.g.
> Hadoop bla –D dfs.blocksize= 134217728)
>
>
>
> Regards
>
>
>
> From: David Sinclair [mailto:dsinclair@chariotsolutions.com]
> Sent: Friday, January 03, 2014 10:47 AM
> To: user@hadoop.apache.org
> Subject: Re: Block size
>
>
>
> Change the dfs.block.size in hdfs-site.xml to be the value you would like if
> you want to have all new files have a different block size.
>
>
>
> On Fri, Jan 3, 2014 at 11:37 AM, Kurt Moesky <ku...@gmail.com> wrote:
>
> I see the default block size for HDFS is 64 MB, is this a value that can be
> changed easily?
>
>
--
Harsh J
Re: Block size
Posted by Harsh J <ha...@cloudera.com>.
XG,
The newer default is 128 MB [HDFS-4053]. The minimum, however, can be
as low as io.bytes.per.checksum (default: 512 bytes) if the user so
wishes it. To administratively set a limit to prevent low values from
being used, see the config introduced via HDFS-4305.
On Sat, Jan 4, 2014 at 11:38 AM, Zhao, Xiaoguang
<Xi...@honeywell.com> wrote:
> As I am new to hdfs, I was told that the minimize block size is 64M, is it
> correct?
>
> XG
>
> 在 2014年1月4日，3:12，"German Florez-Larrahondo" <ge...@samsung.com> 写道：
>
> Also note that the block size in recent releases is actually called
> “dfs.blocksize” as opposed to “dfs.block.size”, and that you can set it per
> job as well. In that scenario, just pass it as an argument to your job (e.g.
> Hadoop bla –D dfs.blocksize= 134217728)
>
>
>
> Regards
>
>
>
> From: David Sinclair [mailto:dsinclair@chariotsolutions.com]
> Sent: Friday, January 03, 2014 10:47 AM
> To: user@hadoop.apache.org
> Subject: Re: Block size
>
>
>
> Change the dfs.block.size in hdfs-site.xml to be the value you would like if
> you want to have all new files have a different block size.
>
>
>
> On Fri, Jan 3, 2014 at 11:37 AM, Kurt Moesky <ku...@gmail.com> wrote:
>
> I see the default block size for HDFS is 64 MB, is this a value that can be
> changed easily?
>
>
--
Harsh J
Re: Block size
Posted by Harsh J <ha...@cloudera.com>.
XG,
The newer default is 128 MB [HDFS-4053]. The minimum, however, can be
as low as io.bytes.per.checksum (default: 512 bytes) if the user so
wishes it. To administratively set a limit to prevent low values from
being used, see the config introduced via HDFS-4305.
On Sat, Jan 4, 2014 at 11:38 AM, Zhao, Xiaoguang
<Xi...@honeywell.com> wrote:
> As I am new to hdfs, I was told that the minimize block size is 64M, is it
> correct?
>
> XG
>
> 在 2014年1月4日，3:12，"German Florez-Larrahondo" <ge...@samsung.com> 写道：
>
> Also note that the block size in recent releases is actually called
> “dfs.blocksize” as opposed to “dfs.block.size”, and that you can set it per
> job as well. In that scenario, just pass it as an argument to your job (e.g.
> Hadoop bla –D dfs.blocksize= 134217728)
>
>
>
> Regards
>
>
>
> From: David Sinclair [mailto:dsinclair@chariotsolutions.com]
> Sent: Friday, January 03, 2014 10:47 AM
> To: user@hadoop.apache.org
> Subject: Re: Block size
>
>
>
> Change the dfs.block.size in hdfs-site.xml to be the value you would like if
> you want to have all new files have a different block size.
>
>
>
> On Fri, Jan 3, 2014 at 11:37 AM, Kurt Moesky <ku...@gmail.com> wrote:
>
> I see the default block size for HDFS is 64 MB, is this a value that can be
> changed easily?
>
>
--
Harsh J
Re: Block size
Posted by "Zhao, Xiaoguang" <Xi...@Honeywell.com>.
As I am new to hdfs, I was told that the minimize block size is 64M, is it correct?
XG
�� 2014��1��4�գ�3:12��"German Florez-Larrahondo" <ge...@samsung.com>> д����
Also note that the block size in recent releases is actually called ��dfs.blocksize�� as opposed to ��dfs.block.size��, and that you can set it per job as well. In that scenario, just pass it as an argument to your job (e.g. Hadoop bla �CD dfs.blocksize= 134217728)
Regards
From: David Sinclair [mailto:dsinclair@chariotsolutions.com]
Sent: Friday, January 03, 2014 10:47 AM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: Block size
Change the dfs.block.size in hdfs-site.xml to be the value you would like if you want to have all new files have a different block size.
On Fri, Jan 3, 2014 at 11:37 AM, Kurt Moesky <ku...@gmail.com>> wrote:
I see the default block size for HDFS is 64 MB, is this a value that can be changed easily?
Re: Block size
Posted by "Zhao, Xiaoguang" <Xi...@Honeywell.com>.
As I am new to hdfs, I was told that the minimize block size is 64M, is it correct?
XG
�� 2014��1��4�գ�3:12��"German Florez-Larrahondo" <ge...@samsung.com>> д����
Also note that the block size in recent releases is actually called ��dfs.blocksize�� as opposed to ��dfs.block.size��, and that you can set it per job as well. In that scenario, just pass it as an argument to your job (e.g. Hadoop bla �CD dfs.blocksize= 134217728)
Regards
From: David Sinclair [mailto:dsinclair@chariotsolutions.com]
Sent: Friday, January 03, 2014 10:47 AM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: Block size
Change the dfs.block.size in hdfs-site.xml to be the value you would like if you want to have all new files have a different block size.
On Fri, Jan 3, 2014 at 11:37 AM, Kurt Moesky <ku...@gmail.com>> wrote:
I see the default block size for HDFS is 64 MB, is this a value that can be changed easily?
Re: Block size
Posted by "Zhao, Xiaoguang" <Xi...@Honeywell.com>.
As I am new to hdfs, I was told that the minimize block size is 64M, is it correct?
XG
�� 2014��1��4�գ�3:12��"German Florez-Larrahondo" <ge...@samsung.com>> д����
Also note that the block size in recent releases is actually called ��dfs.blocksize�� as opposed to ��dfs.block.size��, and that you can set it per job as well. In that scenario, just pass it as an argument to your job (e.g. Hadoop bla �CD dfs.blocksize= 134217728)
Regards
From: David Sinclair [mailto:dsinclair@chariotsolutions.com]
Sent: Friday, January 03, 2014 10:47 AM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: Block size
Change the dfs.block.size in hdfs-site.xml to be the value you would like if you want to have all new files have a different block size.
On Fri, Jan 3, 2014 at 11:37 AM, Kurt Moesky <ku...@gmail.com>> wrote:
I see the default block size for HDFS is 64 MB, is this a value that can be changed easily?
Re: Block size
Posted by "Zhao, Xiaoguang" <Xi...@Honeywell.com>.
As I am new to hdfs, I was told that the minimize block size is 64M, is it correct?
XG
�� 2014��1��4�գ�3:12��"German Florez-Larrahondo" <ge...@samsung.com>> д����
Also note that the block size in recent releases is actually called ��dfs.blocksize�� as opposed to ��dfs.block.size��, and that you can set it per job as well. In that scenario, just pass it as an argument to your job (e.g. Hadoop bla �CD dfs.blocksize= 134217728)
Regards
From: David Sinclair [mailto:dsinclair@chariotsolutions.com]
Sent: Friday, January 03, 2014 10:47 AM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: Block size
Change the dfs.block.size in hdfs-site.xml to be the value you would like if you want to have all new files have a different block size.
On Fri, Jan 3, 2014 at 11:37 AM, Kurt Moesky <ku...@gmail.com>> wrote:
I see the default block size for HDFS is 64 MB, is this a value that can be changed easily?
RE: Block size
Posted by German Florez-Larrahondo <ge...@samsung.com>.
Also note that the block size in recent releases is actually called
"dfs.blocksize" as opposed to "dfs.block.size", and that you can set it per
job as well. In that scenario, just pass it as an argument to your job (e.g.
Hadoop bla -D dfs.blocksize= 134217728)
Regards
From: David Sinclair [mailto:dsinclair@chariotsolutions.com]
Sent: Friday, January 03, 2014 10:47 AM
To: user@hadoop.apache.org
Subject: Re: Block size
Change the dfs.block.size in hdfs-site.xml to be the value you would like if
you want to have all new files have a different block size.
On Fri, Jan 3, 2014 at 11:37 AM, Kurt Moesky <ku...@gmail.com> wrote:
I see the default block size for HDFS is 64 MB, is this a value that can be
changed easily?
RE: Block size
Posted by German Florez-Larrahondo <ge...@samsung.com>.
Also note that the block size in recent releases is actually called
"dfs.blocksize" as opposed to "dfs.block.size", and that you can set it per
job as well. In that scenario, just pass it as an argument to your job (e.g.
Hadoop bla -D dfs.blocksize= 134217728)
Regards
From: David Sinclair [mailto:dsinclair@chariotsolutions.com]
Sent: Friday, January 03, 2014 10:47 AM
To: user@hadoop.apache.org
Subject: Re: Block size
Change the dfs.block.size in hdfs-site.xml to be the value you would like if
you want to have all new files have a different block size.
On Fri, Jan 3, 2014 at 11:37 AM, Kurt Moesky <ku...@gmail.com> wrote:
I see the default block size for HDFS is 64 MB, is this a value that can be
changed easily?
RE: Block size
Posted by German Florez-Larrahondo <ge...@samsung.com>.
Also note that the block size in recent releases is actually called
"dfs.blocksize" as opposed to "dfs.block.size", and that you can set it per
job as well. In that scenario, just pass it as an argument to your job (e.g.
Hadoop bla -D dfs.blocksize= 134217728)
Regards
From: David Sinclair [mailto:dsinclair@chariotsolutions.com]
Sent: Friday, January 03, 2014 10:47 AM
To: user@hadoop.apache.org
Subject: Re: Block size
Change the dfs.block.size in hdfs-site.xml to be the value you would like if
you want to have all new files have a different block size.
On Fri, Jan 3, 2014 at 11:37 AM, Kurt Moesky <ku...@gmail.com> wrote:
I see the default block size for HDFS is 64 MB, is this a value that can be
changed easily?
RE: Block size
Posted by German Florez-Larrahondo <ge...@samsung.com>.
Also note that the block size in recent releases is actually called
"dfs.blocksize" as opposed to "dfs.block.size", and that you can set it per
job as well. In that scenario, just pass it as an argument to your job (e.g.
Hadoop bla -D dfs.blocksize= 134217728)
Regards
From: David Sinclair [mailto:dsinclair@chariotsolutions.com]
Sent: Friday, January 03, 2014 10:47 AM
To: user@hadoop.apache.org
Subject: Re: Block size
Change the dfs.block.size in hdfs-site.xml to be the value you would like if
you want to have all new files have a different block size.
On Fri, Jan 3, 2014 at 11:37 AM, Kurt Moesky <ku...@gmail.com> wrote:
I see the default block size for HDFS is 64 MB, is this a value that can be
changed easily?
Re: Block size
Posted by David Sinclair <ds...@chariotsolutions.com>.
Change the dfs.block.size in hdfs-site.xml to be the value you would like
if you want to have all new files have a different block size.
On Fri, Jan 3, 2014 at 11:37 AM, Kurt Moesky <ku...@gmail.com> wrote:
> I see the default block size for HDFS is 64 MB, is this a value that can
> be changed easily?
>
Re: Block size
Posted by Shahab Yunus <sh...@gmail.com>.
Yes it can. It is a configurable property. The exact name might differ
depending on the version though.
Read the details here:
https://hadoop.apache.org/docs/current2/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml
http://books.google.com/books?id=H3mvcxPeUfwC&pg=PA183&lpg=PA183&dq=change+hadoop+default+block+size&source=bl&ots=pXxPwd2Hv8&sig=xbcUaJ10zOkLqOPw4u35ZTRPCVc&hl=en&sa=X&ei=gujGUs-eK4ulsASMi4GIDw&ved=0CFYQ6AEwAw#v=onepage&q=change%20hadoop%20default%20block%20size&f=false
You can do it programmatically as well.
http://stackoverflow.com/questions/2669800/changing-the-block-size-of-a-dfs-file-in-hadoop
Changing for an existing file:
http://stackoverflow.com/questions/7320625/how-to-dynamic-change-existing-files-block-size-in-hadoop
Regards,
Shahab
On Fri, Jan 3, 2014 at 11:37 AM, Kurt Moesky <ku...@gmail.com> wrote:
> I see the default block size for HDFS is 64 MB, is this a value that can
> be changed easily?
>
Re: Block size
Posted by David Sinclair <ds...@chariotsolutions.com>.
Change the dfs.block.size in hdfs-site.xml to be the value you would like
if you want to have all new files have a different block size.
On Fri, Jan 3, 2014 at 11:37 AM, Kurt Moesky <ku...@gmail.com> wrote:
> I see the default block size for HDFS is 64 MB, is this a value that can
> be changed easily?
>
Re: Block size
Posted by David Sinclair <ds...@chariotsolutions.com>.
Change the dfs.block.size in hdfs-site.xml to be the value you would like
if you want to have all new files have a different block size.
On Fri, Jan 3, 2014 at 11:37 AM, Kurt Moesky <ku...@gmail.com> wrote:
> I see the default block size for HDFS is 64 MB, is this a value that can
> be changed easily?
>
Re: Block size
Posted by Shahab Yunus <sh...@gmail.com>.
Yes it can. It is a configurable property. The exact name might differ
depending on the version though.
Read the details here:
https://hadoop.apache.org/docs/current2/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml
http://books.google.com/books?id=H3mvcxPeUfwC&pg=PA183&lpg=PA183&dq=change+hadoop+default+block+size&source=bl&ots=pXxPwd2Hv8&sig=xbcUaJ10zOkLqOPw4u35ZTRPCVc&hl=en&sa=X&ei=gujGUs-eK4ulsASMi4GIDw&ved=0CFYQ6AEwAw#v=onepage&q=change%20hadoop%20default%20block%20size&f=false
You can do it programmatically as well.
http://stackoverflow.com/questions/2669800/changing-the-block-size-of-a-dfs-file-in-hadoop
Changing for an existing file:
http://stackoverflow.com/questions/7320625/how-to-dynamic-change-existing-files-block-size-in-hadoop
Regards,
Shahab
On Fri, Jan 3, 2014 at 11:37 AM, Kurt Moesky <ku...@gmail.com> wrote:
> I see the default block size for HDFS is 64 MB, is this a value that can
> be changed easily?
>
Re: Block size
Posted by Shahab Yunus <sh...@gmail.com>.
Yes it can. It is a configurable property. The exact name might differ
depending on the version though.
Read the details here:
https://hadoop.apache.org/docs/current2/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml
http://books.google.com/books?id=H3mvcxPeUfwC&pg=PA183&lpg=PA183&dq=change+hadoop+default+block+size&source=bl&ots=pXxPwd2Hv8&sig=xbcUaJ10zOkLqOPw4u35ZTRPCVc&hl=en&sa=X&ei=gujGUs-eK4ulsASMi4GIDw&ved=0CFYQ6AEwAw#v=onepage&q=change%20hadoop%20default%20block%20size&f=false
You can do it programmatically as well.
http://stackoverflow.com/questions/2669800/changing-the-block-size-of-a-dfs-file-in-hadoop
Changing for an existing file:
http://stackoverflow.com/questions/7320625/how-to-dynamic-change-existing-files-block-size-in-hadoop
Regards,
Shahab
On Fri, Jan 3, 2014 at 11:37 AM, Kurt Moesky <ku...@gmail.com> wrote:
> I see the default block size for HDFS is 64 MB, is this a value that can
> be changed easily?
>
Re: Block size
Posted by Shahab Yunus <sh...@gmail.com>.
Yes it can. It is a configurable property. The exact name might differ
depending on the version though.
Read the details here:
https://hadoop.apache.org/docs/current2/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml
http://books.google.com/books?id=H3mvcxPeUfwC&pg=PA183&lpg=PA183&dq=change+hadoop+default+block+size&source=bl&ots=pXxPwd2Hv8&sig=xbcUaJ10zOkLqOPw4u35ZTRPCVc&hl=en&sa=X&ei=gujGUs-eK4ulsASMi4GIDw&ved=0CFYQ6AEwAw#v=onepage&q=change%20hadoop%20default%20block%20size&f=false
You can do it programmatically as well.
http://stackoverflow.com/questions/2669800/changing-the-block-size-of-a-dfs-file-in-hadoop
Changing for an existing file:
http://stackoverflow.com/questions/7320625/how-to-dynamic-change-existing-files-block-size-in-hadoop
Regards,
Shahab
On Fri, Jan 3, 2014 at 11:37 AM, Kurt Moesky <ku...@gmail.com> wrote:
> I see the default block size for HDFS is 64 MB, is this a value that can
> be changed easily?
>
Re: Block size
Posted by David Sinclair <ds...@chariotsolutions.com>.
Change the dfs.block.size in hdfs-site.xml to be the value you would like
if you want to have all new files have a different block size.
On Fri, Jan 3, 2014 at 11:37 AM, Kurt Moesky <ku...@gmail.com> wrote:
> I see the default block size for HDFS is 64 MB, is this a value that can
> be changed easily?
>