Block creation in HDFS

[Abhishek Das <ab...@gmail.com>]

Hi,

I am new in this group. I had a question regarding block creation in HDFS.
By default the file is split into multiple blocks of size equal to block
size. I need to introduce new block creation policy into the system. In my
case the main file is divided into multiple partitions. My goal is to
create the blocks where data is represented from each partition of the
file. Is it possible to introduce the new policy ? If yes, what would the
starting point in the code I should look at.

Regards,
Abhishek Das

RE: Block creation in HDFS

["Gangumalla, Uma" <um...@intel.com>]

Hi,

HDFS does store the data how you writing to it. It will not organize the data. HDFS has flexibility I terms of placements.
If you want to write in this fashion bunch of blocks should be allocated once and client should write all of them based on you portion. Which is sounding something similar to striping approach. Which is not supported right now in HDFS. It is being developed with erasure codes branch. HDFS-7285. Correct me if I misunderstood ur needs here.

Regards,
Uma

-----Original Message-----
From: Abhishek Das [mailto:abhishek.besu@gmail.com] 
Sent: Tuesday, February 17, 2015 11:14 PM
To: hdfs-dev
Subject: Re: Block creation in HDFS

Hi,

Thanks Vinay for your response. I dont need blocks of variable size. But setting only the block size probably wont help in my case. Let me give an example to explain what I am trying to do.

Let say the main file has 12 integers 1 to 12. The block size is such that each block will have 3 integers. Now if I ask hdfs to create the blocks, it would create 4 blocks - first one would have 1-3, second one would have 4-7. According to my requirement, the data in the main file is partitioned into 3 clusters. (1,2,3,4), (5,6,7,8) and (9,10,11,12). Now when the blocks will be created, I need data from all partitions get represented in each block. So in this case, the first block would have (1,5,9), second one would have (2,6,10) etc... So i want to change how the data is allocated in each of the blocks.

Is it feasible to change  the default block creation policy in current implementation?

Regards,
Abhishek Das

On Tue, Feb 17, 2015 at 2:25 AM, Vinayakumar B <vi...@apache.org>
wrote:

> Hi abhishek,
> Is Your partitions of same sizes? If yes, then you can set that as 
> block size.
>
> If not you can use the latest feature.. variable block size.
> To verify your use case.
> You can close the current block after each partition data is written 
> and append to new block for new partition data.
> This feature is not yet available in any of the release. Hope to see 
> in future 2.7 release. As of now you can verify in any of the 
> trunk/branch-2 builds.
>
> Hope this helps.
>
> -Vinay
> On Feb 17, 2015 8:30 AM, "Abhishek Das" <ab...@gmail.com> wrote:
>
> > Hi,
> >
> > I am new in this group. I had a question regarding block creation in
> HDFS.
> > By default the file is split into multiple blocks of size equal to 
> > block size. I need to introduce new block creation policy into the 
> > system. In
> my
> > case the main file is divided into multiple partitions. My goal is 
> > to create the blocks where data is represented from each partition 
> > of the file. Is it possible to introduce the new policy ? If yes, 
> > what would the starting point in the code I should look at.
> >
> > Regards,
> > Abhishek Das
> >
>

Re: Block creation in HDFS

[Abhishek Das <ab...@gmail.com>]

Hi,

Thanks Vinay for your response. I dont need blocks of variable size. But
setting only the block size probably wont help in my case. Let me give an
example to explain what I am trying to do.

Let say the main file has 12 integers 1 to 12. The block size is such that
each block will have 3 integers. Now if I ask hdfs to create the blocks, it
would create 4 blocks - first one would have 1-3, second one would have
4-7. According to my requirement, the data in the main file is partitioned
into 3 clusters. (1,2,3,4), (5,6,7,8) and (9,10,11,12). Now when the blocks
will be created, I need data from all partitions get represented in each
block. So in this case, the first block would have (1,5,9), second one
would have (2,6,10) etc... So i want to change how the data is allocated in
each of the blocks.

Is it feasible to change  the default block creation policy in current
implementation?

Regards,
Abhishek Das

On Tue, Feb 17, 2015 at 2:25 AM, Vinayakumar B <vi...@apache.org>
wrote:

> Hi abhishek,
> Is Your partitions of same sizes? If yes, then you can set that as block
> size.
>
> If not you can use the latest feature.. variable block size.
> To verify your use case.
> You can close the current block after each partition data is written and
> append to new block for new partition data.
> This feature is not yet available in any of the release. Hope to see in
> future 2.7 release. As of now you can verify in any of the trunk/branch-2
> builds.
>
> Hope this helps.
>
> -Vinay
> On Feb 17, 2015 8:30 AM, "Abhishek Das" <ab...@gmail.com> wrote:
>
> > Hi,
> >
> > I am new in this group. I had a question regarding block creation in
> HDFS.
> > By default the file is split into multiple blocks of size equal to block
> > size. I need to introduce new block creation policy into the system. In
> my
> > case the main file is divided into multiple partitions. My goal is to
> > create the blocks where data is represented from each partition of the
> > file. Is it possible to introduce the new policy ? If yes, what would the
> > starting point in the code I should look at.
> >
> > Regards,
> > Abhishek Das
> >
>

Re: Block creation in HDFS

[Vinayakumar B <vi...@apache.org>]

Hi abhishek,
Is Your partitions of same sizes? If yes, then you can set that as block
size.

If not you can use the latest feature.. variable block size.
To verify your use case.
You can close the current block after each partition data is written and
append to new block for new partition data.
This feature is not yet available in any of the release. Hope to see in
future 2.7 release. As of now you can verify in any of the trunk/branch-2
builds.

Hope this helps.

-Vinay
On Feb 17, 2015 8:30 AM, "Abhishek Das" <ab...@gmail.com> wrote:

> Hi,
>
> I am new in this group. I had a question regarding block creation in HDFS.
> By default the file is split into multiple blocks of size equal to block
> size. I need to introduce new block creation policy into the system. In my
> case the main file is divided into multiple partitions. My goal is to
> create the blocks where data is represented from each partition of the
> file. Is it possible to introduce the new policy ? If yes, what would the
> starting point in the code I should look at.
>
> Regards,
> Abhishek Das
>