You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Pallavi Palleti <pa...@corp.aol.com> on 2009/08/11 09:48:51 UTC
File is closed but data is not visible
Hi all,
We have an application where we pull logs from an external server(far apart from hadoop cluster) to hadoop cluster. Sometimes, we could see huge delay (of 1 hour or more) in actually seeing the data in HDFS though the file has been closed and the variable is set to null from the external application.I was in the impression that when I close the file, the data gets reflected in hadoop cluster. Now, in this situation, it is even more complicated to handle write failures as it is giving false impression to the client that data has been written to HDFS. Kindly clarify if my perception is correct. If yes, Could some one tell me what is causing the delay in actually showing the data. During those cases, how can we tackle write failures (due to some temporary issues like data node not available, disk is full) as there is no way, we can figure out the failure at the client side?
Thanks
Pallavi
Re: File is closed but data is not visible
Posted by Raghu Angadi <ra...@yahoo-inc.com>.
Your assumption is correct. When you close the file, others can read the
data. There is no delay expected before the data is visible. If there is
an error either write() or close() would throw an error.
When you say data is not visible do you mean readers can not see the
file or can not see the data? Is it guaranteed that readers open the
file _after_ close returns on the writer?
Raghu.
Palleti, Pallavi wrote:
> Hi Jason,
>
> Apologies for missing version information in my previous mail. I am
> using hadoop-0.18.3. I am getting FSDataOutputStream object using
> fs.create(new Path(some_file_name)), where fs is FileSystem object. And,
> I am closing the file using close().
>
> Thanks
> Pallavi
>
> -----Original Message-----
> From: Jason Venner [mailto:jason.hadoop@gmail.com]
> Sent: Tuesday, August 11, 2009 6:24 PM
> To: common-user@hadoop.apache.org
> Subject: Re: File is closed but data is not visible
>
> Please provide information on what version of hadoop you are using and
> the
> method of opening and closing the file.
>
>
> On Tue, Aug 11, 2009 at 12:48 AM, Pallavi Palleti <
> pallavi.palleti@corp.aol.com> wrote:
>
>> Hi all,
>>
>> We have an application where we pull logs from an external server(far
> apart
>> from hadoop cluster) to hadoop cluster. Sometimes, we could see huge
> delay
>> (of 1 hour or more) in actually seeing the data in HDFS though the
> file has
>> been closed and the variable is set to null from the external
> application.I
>> was in the impression that when I close the file, the data gets
> reflected in
>> hadoop cluster. Now, in this situation, it is even more complicated to
>> handle write failures as it is giving false impression to the client
> that
>> data has been written to HDFS. Kindly clarify if my perception is
> correct.
>> If yes, Could some one tell me what is causing the delay in actually
> showing
>> the data. During those cases, how can we tackle write failures (due to
> some
>> temporary issues like data node not available, disk is full) as there
> is no
>> way, we can figure out the failure at the client side?
>>
>> Thanks
>> Pallavi
>>
>
>
>
RE: File is closed but data is not visible
Posted by "Palleti, Pallavi" <pa...@corp.aol.com>.
Hi Jason,
Apologies for missing version information in my previous mail. I am
using hadoop-0.18.3. I am getting FSDataOutputStream object using
fs.create(new Path(some_file_name)), where fs is FileSystem object. And,
I am closing the file using close().
Thanks
Pallavi
-----Original Message-----
From: Jason Venner [mailto:jason.hadoop@gmail.com]
Sent: Tuesday, August 11, 2009 6:24 PM
To: common-user@hadoop.apache.org
Subject: Re: File is closed but data is not visible
Please provide information on what version of hadoop you are using and
the
method of opening and closing the file.
On Tue, Aug 11, 2009 at 12:48 AM, Pallavi Palleti <
pallavi.palleti@corp.aol.com> wrote:
> Hi all,
>
> We have an application where we pull logs from an external server(far
apart
> from hadoop cluster) to hadoop cluster. Sometimes, we could see huge
delay
> (of 1 hour or more) in actually seeing the data in HDFS though the
file has
> been closed and the variable is set to null from the external
application.I
> was in the impression that when I close the file, the data gets
reflected in
> hadoop cluster. Now, in this situation, it is even more complicated to
> handle write failures as it is giving false impression to the client
that
> data has been written to HDFS. Kindly clarify if my perception is
correct.
> If yes, Could some one tell me what is causing the delay in actually
showing
> the data. During those cases, how can we tackle write failures (due to
some
> temporary issues like data node not available, disk is full) as there
is no
> way, we can figure out the failure at the client side?
>
> Thanks
> Pallavi
>
--
Pro Hadoop, a book to guide you from beginner to hadoop mastery,
http://www.amazon.com/dp/1430219424?tag=jewlerymall
www.prohadoopbook.com a community for Hadoop Professionals
Re: File is closed but data is not visible
Posted by Jason Venner <ja...@gmail.com>.
Please provide information on what version of hadoop you are using and the
method of opening and closing the file.
On Tue, Aug 11, 2009 at 12:48 AM, Pallavi Palleti <
pallavi.palleti@corp.aol.com> wrote:
> Hi all,
>
> We have an application where we pull logs from an external server(far apart
> from hadoop cluster) to hadoop cluster. Sometimes, we could see huge delay
> (of 1 hour or more) in actually seeing the data in HDFS though the file has
> been closed and the variable is set to null from the external application.I
> was in the impression that when I close the file, the data gets reflected in
> hadoop cluster. Now, in this situation, it is even more complicated to
> handle write failures as it is giving false impression to the client that
> data has been written to HDFS. Kindly clarify if my perception is correct.
> If yes, Could some one tell me what is causing the delay in actually showing
> the data. During those cases, how can we tackle write failures (due to some
> temporary issues like data node not available, disk is full) as there is no
> way, we can figure out the failure at the client side?
>
> Thanks
> Pallavi
>
--
Pro Hadoop, a book to guide you from beginner to hadoop mastery,
http://www.amazon.com/dp/1430219424?tag=jewlerymall
www.prohadoopbook.com a community for Hadoop Professionals