You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@chukwa.apache.org by "Ari Rabkin (JIRA)" <ji...@apache.org> on 2009/06/19 22:48:07 UTC
[jira] Created: (CHUKWA-317) cleaner support for archiving chunks
cleaner support for archiving chunks
------------------------------------
Key: CHUKWA-317
URL: https://issues.apache.org/jira/browse/CHUKWA-317
Project: Hadoop Chukwa
Issue Type: Improvement
Components: Data Processors
Affects Versions: 0.3.0
Reporter: Ari Rabkin
Assignee: Ari Rabkin
Priority: Critical
Fix For: 0.3.0
Right now, the demux and archiving processing phases are tied together. It should be possible to do archiving without running demux, by setting a single option.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-317) cleaner support for archiving chunks
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ari Rabkin updated CHUKWA-317:
------------------------------
Fix Version/s: 0.5.0
(was: 0.4.0)
> cleaner support for archiving chunks
> ------------------------------------
>
> Key: CHUKWA-317
> URL: https://issues.apache.org/jira/browse/CHUKWA-317
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: Data Processors
> Affects Versions: 0.3.0
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Priority: Minor
> Fix For: 0.5.0
>
>
> Right now, the demux and archiving processing phases are tied together. It should be possible to do archiving without running demux, by setting a single option.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-317) cleaner support for archiving chunks
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ari Rabkin updated CHUKWA-317:
------------------------------
Priority: Major (was: Critical)
After talking to Jerome, I see that this isn't so bad currently. Still could be better, but not critical.
> cleaner support for archiving chunks
> ------------------------------------
>
> Key: CHUKWA-317
> URL: https://issues.apache.org/jira/browse/CHUKWA-317
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: Data Processors
> Affects Versions: 0.3.0
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Fix For: 0.3.0
>
>
> Right now, the demux and archiving processing phases are tied together. It should be possible to do archiving without running demux, by setting a single option.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-317) cleaner support for archiving chunks
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12730504#action_12730504 ]
Ari Rabkin commented on CHUKWA-317:
-----------------------------------
The first two of these are resolved, for my purposes, by CHUKWA-346. But configuring the final archive path is still not possible. This should be a fairly simple fix, but it's currently low priority.
> cleaner support for archiving chunks
> ------------------------------------
>
> Key: CHUKWA-317
> URL: https://issues.apache.org/jira/browse/CHUKWA-317
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: Data Processors
> Affects Versions: 0.3.0
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Fix For: 0.3.0
>
>
> Right now, the demux and archiving processing phases are tied together. It should be possible to do archiving without running demux, by setting a single option.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-317) cleaner support for archiving chunks
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ari Rabkin updated CHUKWA-317:
------------------------------
Fix Version/s: (was: 0.3.0)
0.4.0
> cleaner support for archiving chunks
> ------------------------------------
>
> Key: CHUKWA-317
> URL: https://issues.apache.org/jira/browse/CHUKWA-317
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: Data Processors
> Affects Versions: 0.3.0
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Priority: Minor
> Fix For: 0.4.0
>
>
> Right now, the demux and archiving processing phases are tied together. It should be possible to do archiving without running demux, by setting a single option.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-317) cleaner support for archiving chunks
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12722008#action_12722008 ]
Ari Rabkin commented on CHUKWA-317:
-----------------------------------
I think there are two possible approaches.
have a shell option, set in chukwa-env.sh, that processSinkFiles looks at to decide what to run.
Have a conf option in chukwa-demux-conf that the demux jobs look at, and if it's set, they exit immediately.
First option is cleaner, second option keeps all the conf on the java side.
> cleaner support for archiving chunks
> ------------------------------------
>
> Key: CHUKWA-317
> URL: https://issues.apache.org/jira/browse/CHUKWA-317
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: Data Processors
> Affects Versions: 0.3.0
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Priority: Critical
> Fix For: 0.3.0
>
>
> Right now, the demux and archiving processing phases are tied together. It should be possible to do archiving without running demux, by setting a single option.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-317) cleaner support for archiving chunks
Posted by "Jerome Boulon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12722032#action_12722032 ]
Jerome Boulon commented on CHUKWA-317:
--------------------------------------
BTW, processSinkFiles.sh is no longer used ... at least at Yahoo!.
> cleaner support for archiving chunks
> ------------------------------------
>
> Key: CHUKWA-317
> URL: https://issues.apache.org/jira/browse/CHUKWA-317
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: Data Processors
> Affects Versions: 0.3.0
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Priority: Critical
> Fix For: 0.3.0
>
>
> Right now, the demux and archiving processing phases are tied together. It should be possible to do archiving without running demux, by setting a single option.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-317) cleaner support for archiving chunks
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12725454#action_12725454 ]
Ari Rabkin commented on CHUKWA-317:
-----------------------------------
Things that I'd like to fix in archive:
1) output dir should be configurable.
2) ArchiveManager only works if it's downstream from demux, since it has very strong assumptions about structure of input dir. (grouped by day, with specified names).
3) Only runs once per day.
> cleaner support for archiving chunks
> ------------------------------------
>
> Key: CHUKWA-317
> URL: https://issues.apache.org/jira/browse/CHUKWA-317
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: Data Processors
> Affects Versions: 0.3.0
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Fix For: 0.3.0
>
>
> Right now, the demux and archiving processing phases are tied together. It should be possible to do archiving without running demux, by setting a single option.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-317) cleaner support for archiving chunks
Posted by "Jerome Boulon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12722029#action_12722029 ]
Jerome Boulon commented on CHUKWA-317:
--------------------------------------
You should be able to run archive without Demux since the Archive is using the same files as input??
If you don't want demux just don't start it and start only the archiveManager but you should set the configuration to look for the collector dataSink file instead of the demuxManager dataSinkFile.
> cleaner support for archiving chunks
> ------------------------------------
>
> Key: CHUKWA-317
> URL: https://issues.apache.org/jira/browse/CHUKWA-317
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: Data Processors
> Affects Versions: 0.3.0
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Priority: Critical
> Fix For: 0.3.0
>
>
> Right now, the demux and archiving processing phases are tied together. It should be possible to do archiving without running demux, by setting a single option.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-317) cleaner support for archiving chunks
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ari Rabkin updated CHUKWA-317:
------------------------------
Priority: Minor (was: Major)
All that's left of this issue is improving conf options, so priority is low.
> cleaner support for archiving chunks
> ------------------------------------
>
> Key: CHUKWA-317
> URL: https://issues.apache.org/jira/browse/CHUKWA-317
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: Data Processors
> Affects Versions: 0.3.0
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Priority: Minor
> Fix For: 0.3.0
>
>
> Right now, the demux and archiving processing phases are tied together. It should be possible to do archiving without running demux, by setting a single option.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.