You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@chukwa.apache.org by Jaydeep Ayachit <ja...@persistent.co.in> on 2010/11/02 17:48:10 UTC
Data loss on collector side
Hello,
When collector is in middle of committing chunks to sequence file, if hdfs becomes unavailable, the collector bails out. What happens to .chukwa file the collector is working on? This file will not be renamed to .done file. The file as such would become orphan as next time collector starts, it will not carry on with this file.
Is there any process, that looks for orphan files and takes action on them? (like rename to .done)
Thanks,
Jaydeep
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
Re: Data loss on collector side
Posted by Ariel Rabkin <as...@gmail.com>.
https://issues.apache.org/jira/browse/CHUKWA-4 has the design
discussion and code.
--Ari
On Wed, Nov 3, 2010 at 12:10 AM, Jaydeep Ayachit
<ja...@persistent.co.in> wrote:
> Hello,
>
>
>
> Could you please provide more information on this?
>
>
>
> Regards
>
> Jaydeep
>
>
>
>
--
Ari Rabkin asrabkin@gmail.com
UC Berkeley Computer Science Department
RE: Data loss on collector side
Posted by Jaydeep Ayachit <ja...@persistent.co.in>.
Hello,
Could you please provide more information on this?
Regards
Jaydeep
From: Jerome Boulon [mailto:jboulon@netflix.com]
Sent: Tuesday, November 02, 2010 10:23 PM
To: chukwa-user@incubator.apache.org
Subject: Re: Data loss on collector side
Hi,
The HDFS writer is not doing that but the LocalWriter (HDFSMover) is exactly doing this.
/Jerome.
On 11/2/10 9:48 AM, "Jaydeep Ayachit" <ja...@persistent.co.in> wrote:
Hello,
When collector is in middle of committing chunks to sequence file, if hdfs becomes unavailable, the collector bails out. What happens to .chukwa file the collector is working on? This file will not be renamed to .done file. The file as such would become orphan as next time collector starts, it will not carry on with this file.
Is there any process, that looks for orphan files and takes action on them? (like rename to .done)
Thanks,
Jaydeep
DISCLAIMER ========== This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
Re: Data loss on collector side
Posted by Jerome Boulon <jb...@netflix.com>.
Hi,
The HDFS writer is not doing that but the LocalWriter (HDFSMover) is exactly doing this.
/Jerome.
On 11/2/10 9:48 AM, "Jaydeep Ayachit" <ja...@persistent.co.in> wrote:
Hello,
When collector is in middle of committing chunks to sequence file, if hdfs becomes unavailable, the collector bails out. What happens to .chukwa file the collector is working on? This file will not be renamed to .done file. The file as such would become orphan as next time collector starts, it will not carry on with this file.
Is there any process, that looks for orphan files and takes action on them? (like rename to .done)
Thanks,
Jaydeep
DISCLAIMER ========== This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.