You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by Gokulakannan M <go...@huawei.com> on 2010/09/26 14:21:36 UTC
is dfsclient caches the data block to local disk before writing?
Hi,
Is staging still used in hdfs when writing the data? This doubt
arose when I was going through the hdfs documents.
ref :
http://hadoop.apache.org/hdfs/docs/current/hdfs_design.html#Staging
I believe dfsclient does not cache the datablock to local fs (as
the document says) but it does streaming of 64KB packets to the datanode and
caches the
packets of current block only in memory via dataqueue and
ackqueue.
Is the document needs to be corrected or my understanding is
wrong?
Thanks,
Gokul
****************************************************************************
***********
This e-mail and attachments contain confidential information from HUAWEI,
which is intended only for the person or entity whose address is listed
above. Any use of the information contained herein in any way (including,
but not limited to, total or partial disclosure, reproduction, or
dissemination) by persons other than the intended recipient's) is
prohibited. If you receive this e-mail in error, please notify the sender by
phone or email immediately and delete it!
RE: is dfsclient caches the data block to local disk before writing?
Posted by Gokulakannan M <go...@huawei.com>.
Thank you dhruba....
Regards,
Gokul
_____
From: Dhruba Borthakur [mailto:dhruba@gmail.com]
Sent: Monday, September 27, 2010 1:30 AM
To: hdfs-user@hadoop.apache.org; gokulm@huawei.com
Cc: hdfs-dev@hadoop.apache.org
Subject: Re: is dfsclient caches the data block to local disk before
writing?
DFS client does not write the data to local disk first. Instead, it streams
data directly to the datanodes in the write pipeline. I will update the
document.
On Sun, Sep 26, 2010 at 5:21 AM, Gokulakannan M <go...@huawei.com> wrote:
Hi,
Is staging still used in hdfs when writing the data? This doubt
arose when I was going through the hdfs documents.
ref :
http://hadoop.apache.org/hdfs/docs/current/hdfs_design.html#Staging
I believe dfsclient does not cache the datablock to local fs (as
the document says) but it does streaming of 64KB packets to the datanode and
caches the
packets of current block only in memory via dataqueue and
ackqueue.
Is the document needs to be corrected or my understanding is
wrong?
Thanks,
Gokul
****************************************************************************
***********
This e-mail and attachments contain confidential information from HUAWEI,
which is intended only for the person or entity whose address is listed
above. Any use of the information contained herein in any way (including,
but not limited to, total or partial disclosure, reproduction, or
dissemination) by persons other than the intended recipient's) is
prohibited. If you receive this e-mail in error, please notify the sender by
phone or email immediately and delete it!
--
Connect to me at http://www.facebook.com/dhruba
RE: is dfsclient caches the data block to local disk before writing?
Posted by Gokulakannan M <go...@huawei.com>.
Thank you dhruba....
Regards,
Gokul
_____
From: Dhruba Borthakur [mailto:dhruba@gmail.com]
Sent: Monday, September 27, 2010 1:30 AM
To: hdfs-user@hadoop.apache.org; gokulm@huawei.com
Cc: hdfs-dev@hadoop.apache.org
Subject: Re: is dfsclient caches the data block to local disk before
writing?
DFS client does not write the data to local disk first. Instead, it streams
data directly to the datanodes in the write pipeline. I will update the
document.
On Sun, Sep 26, 2010 at 5:21 AM, Gokulakannan M <go...@huawei.com> wrote:
Hi,
Is staging still used in hdfs when writing the data? This doubt
arose when I was going through the hdfs documents.
ref :
http://hadoop.apache.org/hdfs/docs/current/hdfs_design.html#Staging
I believe dfsclient does not cache the datablock to local fs (as
the document says) but it does streaming of 64KB packets to the datanode and
caches the
packets of current block only in memory via dataqueue and
ackqueue.
Is the document needs to be corrected or my understanding is
wrong?
Thanks,
Gokul
****************************************************************************
***********
This e-mail and attachments contain confidential information from HUAWEI,
which is intended only for the person or entity whose address is listed
above. Any use of the information contained herein in any way (including,
but not limited to, total or partial disclosure, reproduction, or
dissemination) by persons other than the intended recipient's) is
prohibited. If you receive this e-mail in error, please notify the sender by
phone or email immediately and delete it!
--
Connect to me at http://www.facebook.com/dhruba
Re: is dfsclient caches the data block to local disk before writing?
Posted by Dhruba Borthakur <dh...@gmail.com>.
DFS client does not write the data to local disk first. Instead, it streams
data directly to the datanodes in the write pipeline. I will update the
document.
On Sun, Sep 26, 2010 at 5:21 AM, Gokulakannan M <go...@huawei.com> wrote:
> Hi,
>
>
>
> Is staging still used in hdfs when writing the data? This doubt
> arose when I was going through the hdfs documents.
>
> ref :
> http://hadoop.apache.org/hdfs/docs/current/hdfs_design.html#Staging
>
>
>
> I believe dfsclient *does not cache the datablock to local fs*(as the document says) but it does streaming of 64KB packets to the datanode
> and caches the
>
> packets of current block only in memory via dataqueue and
> ackqueue.
>
>
>
> Is the document needs to be corrected or my understanding is
> wrong?
>
>
>
> Thanks,
>
> Gokul
>
>
>
>
>
>
>
>
> ***************************************************************************************
> This e-mail and attachments contain confidential information from HUAWEI,
> which is intended only for the person or entity whose address is listed
> above. Any use of the information contained herein in any way (including,
> but not limited to, total or partial disclosure, reproduction, or
> dissemination) by persons other than the intended recipient's) is
> prohibited. If you receive this e-mail in error, please notify the sender by
> phone or email immediately and delete it!
>
>
>
--
Connect to me at http://www.facebook.com/dhruba
Re: is dfsclient caches the data block to local disk before writing?
Posted by Dhruba Borthakur <dh...@gmail.com>.
DFS client does not write the data to local disk first. Instead, it streams
data directly to the datanodes in the write pipeline. I will update the
document.
On Sun, Sep 26, 2010 at 5:21 AM, Gokulakannan M <go...@huawei.com> wrote:
> Hi,
>
>
>
> Is staging still used in hdfs when writing the data? This doubt
> arose when I was going through the hdfs documents.
>
> ref :
> http://hadoop.apache.org/hdfs/docs/current/hdfs_design.html#Staging
>
>
>
> I believe dfsclient *does not cache the datablock to local fs*(as the document says) but it does streaming of 64KB packets to the datanode
> and caches the
>
> packets of current block only in memory via dataqueue and
> ackqueue.
>
>
>
> Is the document needs to be corrected or my understanding is
> wrong?
>
>
>
> Thanks,
>
> Gokul
>
>
>
>
>
>
>
>
> ***************************************************************************************
> This e-mail and attachments contain confidential information from HUAWEI,
> which is intended only for the person or entity whose address is listed
> above. Any use of the information contained herein in any way (including,
> but not limited to, total or partial disclosure, reproduction, or
> dissemination) by persons other than the intended recipient's) is
> prohibited. If you receive this e-mail in error, please notify the sender by
> phone or email immediately and delete it!
>
>
>
--
Connect to me at http://www.facebook.com/dhruba