You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by wang shu <uc...@gmail.com> on 2016/12/14 16:07:04 UTC

Question regarding streamer thread implementation at DFSOutputStream.java

Hi all,

I have a question regarding the streamer thread (Line 382-575)
 implementation at DFSOutputStream.java(http://
grepcode.com/file/repo1.maven.org/maven2/org.apache.hadoop/
hadoop-hdfs/2.7.1/org/apache/hadoop/hdfs/DFSOutputStream.
java#DFSOutputStream.DataStreamer.run%28%29).


What's the logic of line 468-485 and line 535-548, why it check the packet
is the last one twice and the comments are different, one is "
// wait for all data packets have been successfully acked" and another is "
// wait for the close packet has been acked".  What's the close packet?

My understanding of this is:
If a packet is not the last in block, then the packet is sent directly, and
no need to wait for ACK.
If a packet is the last one, then it will first wait for all previous
packet to be ACKed, and then it will send the last packet in this block(
called close packet, I guess?), and then will need wait the last packet to
be ACKed.

Thanks a lot for your help.


Best,

Shu

Re: Question regarding streamer thread implementation at DFSOutputStream.java

Posted by wang shu <uc...@gmail.com>.
Hi Manjunath,

I see. Thanks a lot.

Best,

Shu

2017-01-03 11:47 GMT-06:00 Manjunath A <ma...@gmail.com>:

> Hi Shu,
>
> Apologies for dealy in response.
>
> Close packet is something which is the last packet of the current block
> after writing which , the current block is closed and a new pipeline is
> setup for the next block.
>
> Your understanding of the one.isLastPacketInBlock() is correct.
>
> Regards,
> Manjunath
>
> On Wed, Dec 14, 2016 at 9:37 PM, wang shu <uc...@gmail.com>
> wrote:
>
>> Hi all,
>>
>> I have a question regarding the streamer thread (Line 382-575)
>>  implementation at DFSOutputStream.java(http://gr
>> epcode.com/file/repo1.maven.org/maven2/org.apache.hadoop/had
>> oop-hdfs/2.7.1/org/apache/hadoop/hdfs/DFSOutputStream.java#D
>> FSOutputStream.DataStreamer.run%28%29).
>>
>>
>> What's the logic of line 468-485 and line 535-548, why it check the
>> packet is the last one twice and the comments are different, one is "
>> // wait for all data packets have been successfully acked" and another
>> is "// wait for the close packet has been acked".  What's the close
>> packet?
>>
>> My understanding of this is:
>> If a packet is not the last in block, then the packet is sent directly,
>> and no need to wait for ACK.
>> If a packet is the last one, then it will first wait for all previous
>> packet to be ACKed, and then it will send the last packet in this block(
>> called close packet, I guess?), and then will need wait the last packet to
>> be ACKed.
>>
>> Thanks a lot for your help.
>>
>>
>> Best,
>>
>> Shu
>>
>
>

Re: Question regarding streamer thread implementation at DFSOutputStream.java

Posted by Manjunath A <ma...@gmail.com>.
Hi Shu,

Apologies for dealy in response.

Close packet is something which is the last packet of the current block
after writing which , the current block is closed and a new pipeline is
setup for the next block.

Your understanding of the one.isLastPacketInBlock() is correct.

Regards,
Manjunath

On Wed, Dec 14, 2016 at 9:37 PM, wang shu <uc...@gmail.com> wrote:

> Hi all,
>
> I have a question regarding the streamer thread (Line 382-575)
>  implementation at DFSOutputStream.java(http://gr
> epcode.com/file/repo1.maven.org/maven2/org.apache.hadoop/had
> oop-hdfs/2.7.1/org/apache/hadoop/hdfs/DFSOutputStream.java#
> DFSOutputStream.DataStreamer.run%28%29).
>
>
> What's the logic of line 468-485 and line 535-548, why it check the packet
> is the last one twice and the comments are different, one is "
> // wait for all data packets have been successfully acked" and another is
> "// wait for the close packet has been acked".  What's the close packet?
>
> My understanding of this is:
> If a packet is not the last in block, then the packet is sent directly,
> and no need to wait for ACK.
> If a packet is the last one, then it will first wait for all previous
> packet to be ACKed, and then it will send the last packet in this block(
> called close packet, I guess?), and then will need wait the last packet to
> be ACKed.
>
> Thanks a lot for your help.
>
>
> Best,
>
> Shu
>