You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@chukwa.apache.org by "Jerome Boulon (JIRA)" <ji...@apache.org> on 2009/04/30 18:40:31 UTC
[jira] Created: (CHUKWA-204) Agent side file expiration/removal
Agent side file expiration/removal
----------------------------------
Key: CHUKWA-204
URL: https://issues.apache.org/jira/browse/CHUKWA-204
Project: Hadoop Chukwa
Issue Type: New Feature
Components: data collection, Data Processors
Reporter: Jerome Boulon
Priority: Critical
We have a shell script in order to expire files older than x days.
The idea is to move this to ChukwaAgent in order to reduce the number of daemon running and the overall complexity.
This could be done in a background/low priority Cleanup Thread inside the agent.
The input should be a configuration file with:
- path + fileName + regex (similar to the expiration in our Log4J appender)
- number of days (last accessed/modified? date)
The initial version could be as simple as this but a more advanced version will be needed.
The advanced version should be able to track data completeness based on CHUKWA-203 and only remove file that has been completely sent over or generate alerts.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-204) Agent side file expiration/removal
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ari Rabkin updated CHUKWA-204:
------------------------------
Fix Version/s: 0.2.0
Affects Version/s: 0.2.0
> Agent side file expiration/removal
> ----------------------------------
>
> Key: CHUKWA-204
> URL: https://issues.apache.org/jira/browse/CHUKWA-204
> Project: Hadoop Chukwa
> Issue Type: New Feature
> Components: data collection, Data Processors
> Affects Versions: 0.2.0
> Reporter: Jerome Boulon
> Priority: Critical
> Fix For: 0.2.0
>
>
> We have a shell script in order to expire files older than x days.
> The idea is to move this to ChukwaAgent in order to reduce the number of daemon running and the overall complexity.
> This could be done in a background/low priority Cleanup Thread inside the agent.
> The input should be a configuration file with:
> - path + fileName + regex (similar to the expiration in our Log4J appender)
> - number of days (last accessed/modified? date)
> The initial version could be as simple as this but a more advanced version will be needed.
> The advanced version should be able to track data completeness based on CHUKWA-203 and only remove file that has been completely sent over or generate alerts.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-204) Agent side file expiration/removal
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12711224#action_12711224 ]
Eric Yang commented on CHUKWA-204:
----------------------------------
Yes, this is for 0.2.
> Agent side file expiration/removal
> ----------------------------------
>
> Key: CHUKWA-204
> URL: https://issues.apache.org/jira/browse/CHUKWA-204
> Project: Hadoop Chukwa
> Issue Type: New Feature
> Components: data collection, Data Processors
> Reporter: Jerome Boulon
> Priority: Critical
>
> We have a shell script in order to expire files older than x days.
> The idea is to move this to ChukwaAgent in order to reduce the number of daemon running and the overall complexity.
> This could be done in a background/low priority Cleanup Thread inside the agent.
> The input should be a configuration file with:
> - path + fileName + regex (similar to the expiration in our Log4J appender)
> - number of days (last accessed/modified? date)
> The initial version could be as simple as this but a more advanced version will be needed.
> The advanced version should be able to track data completeness based on CHUKWA-203 and only remove file that has been completely sent over or generate alerts.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-204) Agent side file expiration/removal
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12710262#action_12710262 ]
Ari Rabkin commented on CHUKWA-204:
-----------------------------------
My proposal is to bundle this functionality into an Adaptor -- I believe Adaptors have, or should have, enough control over the Agent process to look through the list of current adaptors, and stop the ones that need stopping. And making this an Adaptor means that we can reuse the existing checkpointing and adaptor configuration mechanisms.
> Agent side file expiration/removal
> ----------------------------------
>
> Key: CHUKWA-204
> URL: https://issues.apache.org/jira/browse/CHUKWA-204
> Project: Hadoop Chukwa
> Issue Type: New Feature
> Components: data collection, Data Processors
> Reporter: Jerome Boulon
> Priority: Critical
>
> We have a shell script in order to expire files older than x days.
> The idea is to move this to ChukwaAgent in order to reduce the number of daemon running and the overall complexity.
> This could be done in a background/low priority Cleanup Thread inside the agent.
> The input should be a configuration file with:
> - path + fileName + regex (similar to the expiration in our Log4J appender)
> - number of days (last accessed/modified? date)
> The initial version could be as simple as this but a more advanced version will be needed.
> The advanced version should be able to track data completeness based on CHUKWA-203 and only remove file that has been completely sent over or generate alerts.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-204) Agent side file expiration/removal
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12719785#action_12719785 ]
Ari Rabkin commented on CHUKWA-204:
-----------------------------------
This is for deleting the file, not the adaptor.
> Agent side file expiration/removal
> ----------------------------------
>
> Key: CHUKWA-204
> URL: https://issues.apache.org/jira/browse/CHUKWA-204
> Project: Hadoop Chukwa
> Issue Type: New Feature
> Components: data collection, Data Processors
> Affects Versions: 0.2.0
> Reporter: Jerome Boulon
> Priority: Critical
>
> We have a shell script in order to expire files older than x days.
> The idea is to move this to ChukwaAgent in order to reduce the number of daemon running and the overall complexity.
> This could be done in a background/low priority Cleanup Thread inside the agent.
> The input should be a configuration file with:
> - path + fileName + regex (similar to the expiration in our Log4J appender)
> - number of days (last accessed/modified? date)
> The initial version could be as simple as this but a more advanced version will be needed.
> The advanced version should be able to track data completeness based on CHUKWA-203 and only remove file that has been completely sent over or generate alerts.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-204) Agent side file expiration/removal
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12711055#action_12711055 ]
Ari Rabkin commented on CHUKWA-204:
-----------------------------------
Am I right in thinking this is a 0.2 issue, not a 0.1.2 issue?
> Agent side file expiration/removal
> ----------------------------------
>
> Key: CHUKWA-204
> URL: https://issues.apache.org/jira/browse/CHUKWA-204
> Project: Hadoop Chukwa
> Issue Type: New Feature
> Components: data collection, Data Processors
> Reporter: Jerome Boulon
> Priority: Critical
>
> We have a shell script in order to expire files older than x days.
> The idea is to move this to ChukwaAgent in order to reduce the number of daemon running and the overall complexity.
> This could be done in a background/low priority Cleanup Thread inside the agent.
> The input should be a configuration file with:
> - path + fileName + regex (similar to the expiration in our Log4J appender)
> - number of days (last accessed/modified? date)
> The initial version could be as simple as this but a more advanced version will be needed.
> The advanced version should be able to track data completeness based on CHUKWA-203 and only remove file that has been completely sent over or generate alerts.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-204) Agent side file expiration/removal
Posted by "Mac Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mac Yang updated CHUKWA-204:
----------------------------
Fix Version/s: (was: 0.2.0)
> Agent side file expiration/removal
> ----------------------------------
>
> Key: CHUKWA-204
> URL: https://issues.apache.org/jira/browse/CHUKWA-204
> Project: Hadoop Chukwa
> Issue Type: New Feature
> Components: data collection, Data Processors
> Affects Versions: 0.2.0
> Reporter: Jerome Boulon
> Priority: Critical
>
> We have a shell script in order to expire files older than x days.
> The idea is to move this to ChukwaAgent in order to reduce the number of daemon running and the overall complexity.
> This could be done in a background/low priority Cleanup Thread inside the agent.
> The input should be a configuration file with:
> - path + fileName + regex (similar to the expiration in our Log4J appender)
> - number of days (last accessed/modified? date)
> The initial version could be as simple as this but a more advanced version will be needed.
> The advanced version should be able to track data completeness based on CHUKWA-203 and only remove file that has been completely sent over or generate alerts.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.