You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@chukwa.apache.org by "Jerome Boulon (JIRA)" <ji...@apache.org> on 2009/04/30 18:30:30 UTC
[jira] Created: (CHUKWA-203) Track data loading from agent
Track data loading from agent
-----------------------------
Key: CHUKWA-203
URL: https://issues.apache.org/jira/browse/CHUKWA-203
Project: Hadoop Chukwa
Issue Type: New Feature
Components: data collection, Data Processors
Reporter: Jerome Boulon
Priority: Critical
Chukwa needs to track progress on all files for completeness reason.
The first step could be to send adaptor information to the backend for postprocess/storage.
This could be done at the same time of the writing checkpoint file by building a chunk and post it to the queue.
In addition to that, we need to track all Add/Remove operations and the final offset for all files, the easiest way to do this will be to generate this information at the beginning and the end of each adaptor.
Based on that, we should be able to:
- track any file from the add to the remove,
- validate that all data has been sent
- track all files' rotation.
- record any permission issue (expiration policy)
- generate alerts
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-203) Track data loading from agent
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12717960#action_12717960 ]
Ari Rabkin commented on CHUKWA-203:
-----------------------------------
I don't entirely understand the scope of this. I had thought our model was that adaptors conceal rotation from stages farther up the line?
I definitely like the idea of issuing an "end of file, adaptor has deregistered" chunk/marker.
> Track data loading from agent
> -----------------------------
>
> Key: CHUKWA-203
> URL: https://issues.apache.org/jira/browse/CHUKWA-203
> Project: Hadoop Chukwa
> Issue Type: New Feature
> Components: data collection, Data Processors
> Reporter: Jerome Boulon
> Priority: Critical
>
> Chukwa needs to track progress on all files for completeness reason.
> The first step could be to send adaptor information to the backend for postprocess/storage.
> This could be done at the same time of the writing checkpoint file by building a chunk and post it to the queue.
> In addition to that, we need to track all Add/Remove operations and the final offset for all files, the easiest way to do this will be to generate this information at the beginning and the end of each adaptor.
> Based on that, we should be able to:
> - track any file from the add to the remove,
> - validate that all data has been sent
> - track all files' rotation.
> - record any permission issue (expiration policy)
> - generate alerts
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.