You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by MOHAMMED IRFANULLA S <m....@huawei.com> on 2010/01/25 08:20:50 UTC
HDFS File read issue
Hi,
I'm using hadoop 0.20.1. I would appreciate any help on the following issue
in HDFS.
User1 has created a file file1.txt and started writing to this file(Writer
thread).
User2 and user3 try to read from this file. But cannot read anything until
atleast one of the blocks is complete. and they cannot read any block under
development. (Reader threads)
Is it possible to block/prevent user2 and User3 from reading the file1.txt
completely until the Writer thread calls close().
If possible, how to achieve it ?
MOHAMMED IRFANULLA S
HUAWEI TECHNOLOGIES CO.,LTD. huawei_logo
Address: Huawei Industrial Base
Bantian Longgang
Shenzhen 518129, P.R.China
www.huawei.com
----------------------------------------------------------------------------
---------------------------------------------------------
This e-mail and its attachments contain confidential information from
HUAWEI, which
is intended only for the person or entity whose address is listed above. Any
use of the
information contained herein in any way (including, but not limited to,
total or partial
disclosure, reproduction, or dissemination) by persons other than the
intended
recipient(s) is prohibited. If you receive this e-mail in error, please
notify the sender by
phone or email immediately and delete it!
Re: HDFS File read issue
Posted by Eric Sammer <er...@lifeless.net>.
On 1/25/10 2:20 AM, MOHAMMED IRFANULLA S wrote:
> Hi,
>
> I'm using hadoop 0.20.1. I would appreciate any help on the following
> issue in HDFS.
>
> User1 has created a file file1.txt and started writing to this
> file(Writer thread).
> User2 and user3 try to read from this file. But cannot read anything
> until atleast one of the blocks is complete. and they cannot read any
> block under development. (Reader threads)
>
> Is it possible to block/prevent user2 and User3 from reading the
> file1.txt completely until the Writer thread calls close().
> If possible, how to achieve it ?
Mohammed:
This is the documented behavior of HDFS with regard to data visibility.
Currently, there is no way to prevent block access from user 2 and 3 in
your scenario until user 1 finishes writing; you'd have to implement it
at a layer higher up than HDFS.
In theory, one can force a sync by calling FSDataOutputStream#sync() but
I think that's still buggy (slated for fix in 0.21.x? - see
HDFS-200[1]). This would trade performance for visibility. I think the
alternative of forcing the readers to block until some event triggered
(after the file is completely written) by the writer is a better plan,
though.
Hope this helps.
[1] - https://issues.apache.org/jira/browse/HDFS-200
--
Eric Sammer
eric@lifeless.net
http://esammer.blogspot.com
RE: HDFS File read issue
Posted by MOHAMMED IRFANULLA S <m....@huawei.com>.
Hi, Jeff. Thanks for your reply.
Is there a way to achieve this apart from using Zookeeper ?
MOHAMMED IRFANULLA S
HUAWEI TECHNOLOGIES CO.,LTD. huawei_logo
Address: Huawei Industrial Base
Bantian Longgang
Shenzhen 518129, P.R.China
www.huawei.com
----------------------------------------------------------------------------
---------------------------------------------------------
This e-mail and its attachments contain confidential information from
HUAWEI, which
is intended only for the person or entity whose address is listed above. Any
use of the
information contained herein in any way (including, but not limited to,
total or partial
disclosure, reproduction, or dissemination) by persons other than the
intended
recipient(s) is prohibited. If you receive this e-mail in error, please
notify the sender by
phone or email immediately and delete it!
_____
From: Jeff Zhang [mailto:zjffdu@gmail.com]
Sent: Monday, January 25, 2010 1:06 PM
To: hdfs-user@hadoop.apache.org; m.irfanulla@huawei.com
Subject: Re: HDFS File read issue
Maybe you can use zookeeper to achieve this
On Mon, Jan 25, 2010 at 3:20 PM, MOHAMMED IRFANULLA S
<m....@huawei.com> wrote:
Hi,
I'm using hadoop 0.20.1.
I would appreciate any help on the following issue in HDFS.
User1 has created a file file1.txt and started writing to this file(Writer
thread).
User2 and user3 try to read from this file. But cannot read anything until
atleast one of the blocks is complete. and they cannot read any block under
development. (Reader threads)
Is it possible to block/prevent user2 and User3 from reading the file1.txt
completely until the Writer thread calls close().
If possible, how to achieve it ?
MOHAMMED IRFANULLA S
HUAWEI TECHNOLOGIES CO.,LTD. huawei_logo
Address: Huawei Industrial Base
Bantian Longgang
Shenzhen 518129, P.R.China
www.huawei.com
----------------------------------------------------------------------------
---------------------------------------------------------
This e-mail and its attachments contain confidential information from
HUAWEI, which
is intended only for the person or entity whose address is listed above. Any
use of the
information contained herein in any way (including, but not limited to,
total or partial
disclosure, reproduction, or dissemination) by persons other than the
intended
recipient(s) is prohibited. If you receive this e-mail in error, please
notify the sender by
phone or email immediately and delete it!
--
Best Regards
Jeff Zhang
Re: HDFS File read issue
Posted by Jeff Zhang <zj...@gmail.com>.
Maybe you can use zookeeper to achieve this
On Mon, Jan 25, 2010 at 3:20 PM, MOHAMMED IRFANULLA S <
m.irfanulla@huawei.com> wrote:
> Hi,
>
> I'm using hadoop 0.20.1.
> I would appreciate any help on the following issue in HDFS.
>
> User1 has created a file file1.txt and started writing to this file(Writer
> thread).
> User2 and user3 try to read from this file. But cannot read anything until
> atleast one of the blocks is complete. and they cannot read any block under
> development. (Reader threads)
>
> Is it possible to block/prevent user2 and User3 from reading the file1.txt
> completely until the Writer thread calls close().
> If possible, how to achieve it ?
>
> MOHAMMED IRFANULLA S
> HUAWEI TECHNOLOGIES CO.,LTD. [image: huawei_logo]
>
>
> Address: Huawei Industrial Base
> Bantian Longgang
> Shenzhen 518129, P.R.China
> www.huawei.com
>
> -------------------------------------------------------------------------------------------------------------------------------------
> This e-mail and its attachments contain confidential information from
> HUAWEI, which
> is intended only for the person or entity whose address is listed above.
> Any use of the
> information contained herein in any way (including, but not limited to,
> total or partial
> disclosure, reproduction, or dissemination) by persons other than the
> intended
> recipient(s) is prohibited. If you receive this e-mail in error, please
> notify the sender by
> phone or email immediately and delete it!
>
>
--
Best Regards
Jeff Zhang