You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by kun yan <ya...@gmail.com> on 2013/09/10 05:31:39 UTC
modify hdfs block size
Hi all
Can I modify HDFS data block size is 32MB, I know the default is 64MB
thanks
--
In the Hadoop world, I am just a novice, explore the entire Hadoop
ecosystem, I hope one day I can contribute their own code
YanBit
yankunhadoop@gmail.com
Re: modify hdfs block size
Posted by Vinayakumar B <vi...@gmail.com>.
You can change it to any size in multiples of 512 bytes by default which is
bytesPerChecksum.
But setting it to lesser values leads to heavy load on cluster and setting
to very high value will not distribute the data. So 64MB or (128MB in
latest trunk.) Is recommended as optimal. Its upto you to decide based on
your usecase.
Regards,
Vinayakumar B
On Sep 10, 2013 9:02 AM, "kun yan" <ya...@gmail.com> wrote:
> Hi all
> Can I modify HDFS data block size is 32MB, I know the default is 64MB
> thanks
>
> --
>
> In the Hadoop world, I am just a novice, explore the entire Hadoop
> ecosystem, I hope one day I can contribute their own code
>
> YanBit
> yankunhadoop@gmail.com
>
>
RE: modify hdfs block size
Posted by Brahma Reddy Battula <br...@huawei.com>.
You can change the block size of existing files with a command like
hadoop distcp -Ddfs.block.size=$[256*1024*1024] /path/to/inputdata /path/to/inputdata-with-largeblocks.
After this command completes, you can remove the original data
________________________________
From: kun yan [yankunhadoop@gmail.com]
Sent: Tuesday, September 10, 2013 12:27 PM
To: user@hadoop.apache.org
Subject: Re: modify hdfs block size
thank your very much
2013/9/10 Harsh J <ha...@cloudera.com>>
You cannot change the blocksize (i.e. merge or split) of an existing
file. You can however change it for newer files, and also download and
re-upload older files again with newer blocksize to change it.
On Tue, Sep 10, 2013 at 9:01 AM, kun yan <ya...@gmail.com>> wrote:
> Hi all
> Can I modify HDFS data block size is 32MB, I know the default is 64MB
> thanks
>
> --
>
> In the Hadoop world, I am just a novice, explore the entire Hadoop
> ecosystem, I hope one day I can contribute their own code
>
> YanBit
> yankunhadoop@gmail.com<ma...@gmail.com>
>
--
Harsh J
--
In the Hadoop world, I am just a novice, explore the entire Hadoop ecosystem, I hope one day I can contribute their own code
YanBit
yankunhadoop@gmail.com<ma...@gmail.com>
RE: modify hdfs block size
Posted by Brahma Reddy Battula <br...@huawei.com>.
You can change the block size of existing files with a command like
hadoop distcp -Ddfs.block.size=$[256*1024*1024] /path/to/inputdata /path/to/inputdata-with-largeblocks.
After this command completes, you can remove the original data
________________________________
From: kun yan [yankunhadoop@gmail.com]
Sent: Tuesday, September 10, 2013 12:27 PM
To: user@hadoop.apache.org
Subject: Re: modify hdfs block size
thank your very much
2013/9/10 Harsh J <ha...@cloudera.com>>
You cannot change the blocksize (i.e. merge or split) of an existing
file. You can however change it for newer files, and also download and
re-upload older files again with newer blocksize to change it.
On Tue, Sep 10, 2013 at 9:01 AM, kun yan <ya...@gmail.com>> wrote:
> Hi all
> Can I modify HDFS data block size is 32MB, I know the default is 64MB
> thanks
>
> --
>
> In the Hadoop world, I am just a novice, explore the entire Hadoop
> ecosystem, I hope one day I can contribute their own code
>
> YanBit
> yankunhadoop@gmail.com<ma...@gmail.com>
>
--
Harsh J
--
In the Hadoop world, I am just a novice, explore the entire Hadoop ecosystem, I hope one day I can contribute their own code
YanBit
yankunhadoop@gmail.com<ma...@gmail.com>
RE: modify hdfs block size
Posted by Brahma Reddy Battula <br...@huawei.com>.
You can change the block size of existing files with a command like
hadoop distcp -Ddfs.block.size=$[256*1024*1024] /path/to/inputdata /path/to/inputdata-with-largeblocks.
After this command completes, you can remove the original data
________________________________
From: kun yan [yankunhadoop@gmail.com]
Sent: Tuesday, September 10, 2013 12:27 PM
To: user@hadoop.apache.org
Subject: Re: modify hdfs block size
thank your very much
2013/9/10 Harsh J <ha...@cloudera.com>>
You cannot change the blocksize (i.e. merge or split) of an existing
file. You can however change it for newer files, and also download and
re-upload older files again with newer blocksize to change it.
On Tue, Sep 10, 2013 at 9:01 AM, kun yan <ya...@gmail.com>> wrote:
> Hi all
> Can I modify HDFS data block size is 32MB, I know the default is 64MB
> thanks
>
> --
>
> In the Hadoop world, I am just a novice, explore the entire Hadoop
> ecosystem, I hope one day I can contribute their own code
>
> YanBit
> yankunhadoop@gmail.com<ma...@gmail.com>
>
--
Harsh J
--
In the Hadoop world, I am just a novice, explore the entire Hadoop ecosystem, I hope one day I can contribute their own code
YanBit
yankunhadoop@gmail.com<ma...@gmail.com>
RE: modify hdfs block size
Posted by Brahma Reddy Battula <br...@huawei.com>.
You can change the block size of existing files with a command like
hadoop distcp -Ddfs.block.size=$[256*1024*1024] /path/to/inputdata /path/to/inputdata-with-largeblocks.
After this command completes, you can remove the original data
________________________________
From: kun yan [yankunhadoop@gmail.com]
Sent: Tuesday, September 10, 2013 12:27 PM
To: user@hadoop.apache.org
Subject: Re: modify hdfs block size
thank your very much
2013/9/10 Harsh J <ha...@cloudera.com>>
You cannot change the blocksize (i.e. merge or split) of an existing
file. You can however change it for newer files, and also download and
re-upload older files again with newer blocksize to change it.
On Tue, Sep 10, 2013 at 9:01 AM, kun yan <ya...@gmail.com>> wrote:
> Hi all
> Can I modify HDFS data block size is 32MB, I know the default is 64MB
> thanks
>
> --
>
> In the Hadoop world, I am just a novice, explore the entire Hadoop
> ecosystem, I hope one day I can contribute their own code
>
> YanBit
> yankunhadoop@gmail.com<ma...@gmail.com>
>
--
Harsh J
--
In the Hadoop world, I am just a novice, explore the entire Hadoop ecosystem, I hope one day I can contribute their own code
YanBit
yankunhadoop@gmail.com<ma...@gmail.com>
Re: modify hdfs block size
Posted by kun yan <ya...@gmail.com>.
thank your very much
2013/9/10 Harsh J <ha...@cloudera.com>
> You cannot change the blocksize (i.e. merge or split) of an existing
> file. You can however change it for newer files, and also download and
> re-upload older files again with newer blocksize to change it.
>
> On Tue, Sep 10, 2013 at 9:01 AM, kun yan <ya...@gmail.com> wrote:
> > Hi all
> > Can I modify HDFS data block size is 32MB, I know the default is 64MB
> > thanks
> >
> > --
> >
> > In the Hadoop world, I am just a novice, explore the entire Hadoop
> > ecosystem, I hope one day I can contribute their own code
> >
> > YanBit
> > yankunhadoop@gmail.com
> >
>
>
>
> --
> Harsh J
>
--
In the Hadoop world, I am just a novice, explore the entire Hadoop
ecosystem, I hope one day I can contribute their own code
YanBit
yankunhadoop@gmail.com
Re: modify hdfs block size
Posted by kun yan <ya...@gmail.com>.
thank your very much
2013/9/10 Harsh J <ha...@cloudera.com>
> You cannot change the blocksize (i.e. merge or split) of an existing
> file. You can however change it for newer files, and also download and
> re-upload older files again with newer blocksize to change it.
>
> On Tue, Sep 10, 2013 at 9:01 AM, kun yan <ya...@gmail.com> wrote:
> > Hi all
> > Can I modify HDFS data block size is 32MB, I know the default is 64MB
> > thanks
> >
> > --
> >
> > In the Hadoop world, I am just a novice, explore the entire Hadoop
> > ecosystem, I hope one day I can contribute their own code
> >
> > YanBit
> > yankunhadoop@gmail.com
> >
>
>
>
> --
> Harsh J
>
--
In the Hadoop world, I am just a novice, explore the entire Hadoop
ecosystem, I hope one day I can contribute their own code
YanBit
yankunhadoop@gmail.com
Re: modify hdfs block size
Posted by kun yan <ya...@gmail.com>.
thank your very much
2013/9/10 Harsh J <ha...@cloudera.com>
> You cannot change the blocksize (i.e. merge or split) of an existing
> file. You can however change it for newer files, and also download and
> re-upload older files again with newer blocksize to change it.
>
> On Tue, Sep 10, 2013 at 9:01 AM, kun yan <ya...@gmail.com> wrote:
> > Hi all
> > Can I modify HDFS data block size is 32MB, I know the default is 64MB
> > thanks
> >
> > --
> >
> > In the Hadoop world, I am just a novice, explore the entire Hadoop
> > ecosystem, I hope one day I can contribute their own code
> >
> > YanBit
> > yankunhadoop@gmail.com
> >
>
>
>
> --
> Harsh J
>
--
In the Hadoop world, I am just a novice, explore the entire Hadoop
ecosystem, I hope one day I can contribute their own code
YanBit
yankunhadoop@gmail.com
Re: modify hdfs block size
Posted by kun yan <ya...@gmail.com>.
thank your very much
2013/9/10 Harsh J <ha...@cloudera.com>
> You cannot change the blocksize (i.e. merge or split) of an existing
> file. You can however change it for newer files, and also download and
> re-upload older files again with newer blocksize to change it.
>
> On Tue, Sep 10, 2013 at 9:01 AM, kun yan <ya...@gmail.com> wrote:
> > Hi all
> > Can I modify HDFS data block size is 32MB, I know the default is 64MB
> > thanks
> >
> > --
> >
> > In the Hadoop world, I am just a novice, explore the entire Hadoop
> > ecosystem, I hope one day I can contribute their own code
> >
> > YanBit
> > yankunhadoop@gmail.com
> >
>
>
>
> --
> Harsh J
>
--
In the Hadoop world, I am just a novice, explore the entire Hadoop
ecosystem, I hope one day I can contribute their own code
YanBit
yankunhadoop@gmail.com
Re: modify hdfs block size
Posted by Harsh J <ha...@cloudera.com>.
You cannot change the blocksize (i.e. merge or split) of an existing
file. You can however change it for newer files, and also download and
re-upload older files again with newer blocksize to change it.
On Tue, Sep 10, 2013 at 9:01 AM, kun yan <ya...@gmail.com> wrote:
> Hi all
> Can I modify HDFS data block size is 32MB, I know the default is 64MB
> thanks
>
> --
>
> In the Hadoop world, I am just a novice, explore the entire Hadoop
> ecosystem, I hope one day I can contribute their own code
>
> YanBit
> yankunhadoop@gmail.com
>
--
Harsh J
Re: modify hdfs block size
Posted by Vinayakumar B <vi...@gmail.com>.
You can change it to any size in multiples of 512 bytes by default which is
bytesPerChecksum.
But setting it to lesser values leads to heavy load on cluster and setting
to very high value will not distribute the data. So 64MB or (128MB in
latest trunk.) Is recommended as optimal. Its upto you to decide based on
your usecase.
Regards,
Vinayakumar B
On Sep 10, 2013 9:02 AM, "kun yan" <ya...@gmail.com> wrote:
> Hi all
> Can I modify HDFS data block size is 32MB, I know the default is 64MB
> thanks
>
> --
>
> In the Hadoop world, I am just a novice, explore the entire Hadoop
> ecosystem, I hope one day I can contribute their own code
>
> YanBit
> yankunhadoop@gmail.com
>
>
Re: modify hdfs block size
Posted by Vinayakumar B <vi...@gmail.com>.
You can change it to any size in multiples of 512 bytes by default which is
bytesPerChecksum.
But setting it to lesser values leads to heavy load on cluster and setting
to very high value will not distribute the data. So 64MB or (128MB in
latest trunk.) Is recommended as optimal. Its upto you to decide based on
your usecase.
Regards,
Vinayakumar B
On Sep 10, 2013 9:02 AM, "kun yan" <ya...@gmail.com> wrote:
> Hi all
> Can I modify HDFS data block size is 32MB, I know the default is 64MB
> thanks
>
> --
>
> In the Hadoop world, I am just a novice, explore the entire Hadoop
> ecosystem, I hope one day I can contribute their own code
>
> YanBit
> yankunhadoop@gmail.com
>
>
Re: modify hdfs block size
Posted by Harsh J <ha...@cloudera.com>.
You cannot change the blocksize (i.e. merge or split) of an existing
file. You can however change it for newer files, and also download and
re-upload older files again with newer blocksize to change it.
On Tue, Sep 10, 2013 at 9:01 AM, kun yan <ya...@gmail.com> wrote:
> Hi all
> Can I modify HDFS data block size is 32MB, I know the default is 64MB
> thanks
>
> --
>
> In the Hadoop world, I am just a novice, explore the entire Hadoop
> ecosystem, I hope one day I can contribute their own code
>
> YanBit
> yankunhadoop@gmail.com
>
--
Harsh J
Re: modify hdfs block size
Posted by Harsh J <ha...@cloudera.com>.
You cannot change the blocksize (i.e. merge or split) of an existing
file. You can however change it for newer files, and also download and
re-upload older files again with newer blocksize to change it.
On Tue, Sep 10, 2013 at 9:01 AM, kun yan <ya...@gmail.com> wrote:
> Hi all
> Can I modify HDFS data block size is 32MB, I know the default is 64MB
> thanks
>
> --
>
> In the Hadoop world, I am just a novice, explore the entire Hadoop
> ecosystem, I hope one day I can contribute their own code
>
> YanBit
> yankunhadoop@gmail.com
>
--
Harsh J
Re: modify hdfs block size
Posted by Vinayakumar B <vi...@gmail.com>.
You can change it to any size in multiples of 512 bytes by default which is
bytesPerChecksum.
But setting it to lesser values leads to heavy load on cluster and setting
to very high value will not distribute the data. So 64MB or (128MB in
latest trunk.) Is recommended as optimal. Its upto you to decide based on
your usecase.
Regards,
Vinayakumar B
On Sep 10, 2013 9:02 AM, "kun yan" <ya...@gmail.com> wrote:
> Hi all
> Can I modify HDFS data block size is 32MB, I know the default is 64MB
> thanks
>
> --
>
> In the Hadoop world, I am just a novice, explore the entire Hadoop
> ecosystem, I hope one day I can contribute their own code
>
> YanBit
> yankunhadoop@gmail.com
>
>
Re: modify hdfs block size
Posted by Harsh J <ha...@cloudera.com>.
You cannot change the blocksize (i.e. merge or split) of an existing
file. You can however change it for newer files, and also download and
re-upload older files again with newer blocksize to change it.
On Tue, Sep 10, 2013 at 9:01 AM, kun yan <ya...@gmail.com> wrote:
> Hi all
> Can I modify HDFS data block size is 32MB, I know the default is 64MB
> thanks
>
> --
>
> In the Hadoop world, I am just a novice, explore the entire Hadoop
> ecosystem, I hope one day I can contribute their own code
>
> YanBit
> yankunhadoop@gmail.com
>
--
Harsh J