HDFS : Question on INodeFileUnderConstruction targets filed ..

[Sangmin Lee <sa...@gmail.com>]

Hi folks,

I wonder why hadoop.dfs.INodeFileUnderConstruction Class has "targets"
field.
Can anyone explain this to me?
any help would be appreciated.

-sangmin

Re: HDFS : Question on INodeFileUnderConstruction targets filed ..

[Sangmin Lee <sa...@gmail.com>]

Dear dhruba,

Thanks you for your kind explanation.
Unfortunately, I failed to locate the code for use of 'targets' as you
described.
I was only able to see "targets" field be used by
dfs.FSEditLog.logOpenFile().
When LeaseMonitor finds a expired lease, it seems to get datanode locations
just from FSNamesystem.blockMaps rather than from the 'targets' field.
I guess I am missing something. Could you shed some more light on this?

I appreciate your help again.

Sangmin


On Fri, Aug 1, 2008 at 1:15 AM, Dhruba Borthakur <dh...@gmail.com> wrote:

> The "targets" field is used to record the block location(s of the last
> allocated block fro a file. It is important to remember this at block
> allocation time itself so that data written to the block can be
> recovered if the client dies before it finished writing to the entire
> block.
>
> Take the case when the client started writing to the 3 datanodes in
> the pipeline for that block. Now, let's assume that the client died
> aftre writing 100 bytes to the first two replicas whereas the third
> replica has only 50 bytes in the block. The namenode holds a lease for
> this client. If the client dies prematurely, the namenode experiences
> a timeout for the client and triggers lease recovery for the file(s)
> that were being written by the client. The namenode has to ensure that
> all the three replicas are consistent with one another, otherwise
> readers of the block might see inconsistent data depending on which
> replica they access. Thus, the namenode has to remember which
> datanode(s) possibly has replicas of that block.
>
> hope this help,
> dhruba
>
> On Thu, Jul 31, 2008 at 12:27 PM, Sangmin Lee <sa...@gmail.com>
> wrote:
> > To clarify my question, what is the need for storing the locations of
> last
> > block without being reported by datanodes?
> >
> > On Thu, Jul 31, 2008 at 3:18 PM, Sangmin Lee <sa...@gmail.com>
> wrote:
> >
> >> Hi folks,
> >>
> >> I wonder why hadoop.dfs.INodeFileUnderConstruction Class has "targets"
> >> field.
> >> Can anyone explain this to me?
> >> any help would be appreciated.
> >>
> >> -sangmin
> >>
> >
>

Re: HDFS : Question on INodeFileUnderConstruction targets filed ..

[Dhruba Borthakur <dh...@gmail.com>]

The "targets" field is used to record the block location(s of the last
allocated block fro a file. It is important to remember this at block
allocation time itself so that data written to the block can be
recovered if the client dies before it finished writing to the entire
block.

Take the case when the client started writing to the 3 datanodes in
the pipeline for that block. Now, let's assume that the client died
aftre writing 100 bytes to the first two replicas whereas the third
replica has only 50 bytes in the block. The namenode holds a lease for
this client. If the client dies prematurely, the namenode experiences
a timeout for the client and triggers lease recovery for the file(s)
that were being written by the client. The namenode has to ensure that
all the three replicas are consistent with one another, otherwise
readers of the block might see inconsistent data depending on which
replica they access. Thus, the namenode has to remember which
datanode(s) possibly has replicas of that block.

hope this help,
dhruba

On Thu, Jul 31, 2008 at 12:27 PM, Sangmin Lee <sa...@gmail.com> wrote:
> To clarify my question, what is the need for storing the locations of last
> block without being reported by datanodes?
>
> On Thu, Jul 31, 2008 at 3:18 PM, Sangmin Lee <sa...@gmail.com> wrote:
>
>> Hi folks,
>>
>> I wonder why hadoop.dfs.INodeFileUnderConstruction Class has "targets"
>> field.
>> Can anyone explain this to me?
>> any help would be appreciated.
>>
>> -sangmin
>>
>

Re: HDFS : Question on INodeFileUnderConstruction targets filed ..

[Sangmin Lee <sa...@gmail.com>]

To clarify my question, what is the need for storing the locations of last
block without being reported by datanodes?

On Thu, Jul 31, 2008 at 3:18 PM, Sangmin Lee <sa...@gmail.com> wrote:

> Hi folks,
>
> I wonder why hadoop.dfs.INodeFileUnderConstruction Class has "targets"
> field.
> Can anyone explain this to me?
> any help would be appreciated.
>
> -sangmin
>