You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@chukwa.apache.org by "Bill Graham (JIRA)" <ji...@apache.org> on 2010/04/19 23:08:49 UTC
[jira] Created: (CHUKWA-477) Support post-demux triggers
Support post-demux triggers
---------------------------
Key: CHUKWA-477
URL: https://issues.apache.org/jira/browse/CHUKWA-477
Project: Hadoop Chukwa
Issue Type: New Feature
Components: Data Processors
Reporter: Bill Graham
Assignee: Bill Graham
Add the ability to trigger an action upon successful completion of a Demux process.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-477) Support post-demux triggers
Posted by "Bill Graham (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bill Graham updated CHUKWA-477:
-------------------------------
Status: Patch Available (was: Open)
> Support post-demux triggers
> ---------------------------
>
> Key: CHUKWA-477
> URL: https://issues.apache.org/jira/browse/CHUKWA-477
> Project: Hadoop Chukwa
> Issue Type: New Feature
> Components: Data Processors
> Reporter: Bill Graham
> Assignee: Bill Graham
> Attachments: CHUKWA-477.1.patch, CHUKWA-477.2.patch, CHUKWA-477.3.patch, CHUKWA-477.4.patch
>
>
> Add the ability to trigger an action upon successful completion of a Demux process.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-477) Support post-demux triggers
Posted by "Bill Graham (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bill Graham updated CHUKWA-477:
-------------------------------
Status: Patch Available (was: Open)
Release Note: Hooks added to trigger actions upon successful completion of a Demux run.
> Support post-demux triggers
> ---------------------------
>
> Key: CHUKWA-477
> URL: https://issues.apache.org/jira/browse/CHUKWA-477
> Project: Hadoop Chukwa
> Issue Type: New Feature
> Components: Data Processors
> Reporter: Bill Graham
> Assignee: Bill Graham
> Attachments: CHUKWA-477.1.patch, CHUKWA-477.2.patch
>
>
> Add the ability to trigger an action upon successful completion of a Demux process.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-477) Support post-demux triggers
Posted by "Bill Graham (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bill Graham updated CHUKWA-477:
-------------------------------
Status: Open (was: Patch Available)
Canceling latest patch 3, since I plan to resubmit with a change to the signature of TriggerAction to accept info regarding the event causing the trigger.
> Support post-demux triggers
> ---------------------------
>
> Key: CHUKWA-477
> URL: https://issues.apache.org/jira/browse/CHUKWA-477
> Project: Hadoop Chukwa
> Issue Type: New Feature
> Components: Data Processors
> Reporter: Bill Graham
> Assignee: Bill Graham
> Attachments: CHUKWA-477.1.patch, CHUKWA-477.2.patch, CHUKWA-477.3.patch
>
>
> Add the ability to trigger an action upon successful completion of a Demux process.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-477) Support post-demux triggers
Posted by "Bill Graham (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bill Graham updated CHUKWA-477:
-------------------------------
Attachment: CHUKWA-477.2.patch
Attaching CHUKWA-477.2.patch, which includes the following changes:
- I've changed the 'load' method to 'execute' in TriggerAction, which makes more sense. It has the same signature though, so we could easily refactor the DataLoaders to implement TriggerAction at some point.
- I modified MoveToRepository to return the HDFS file names that were created as part of the move operation.
- I'm passing the moved files collection into the TriggerAction.
- I added a sample config in chukwa-demux-conf.xml.template to show how to configure a post-demux trigger.
> Support post-demux triggers
> ---------------------------
>
> Key: CHUKWA-477
> URL: https://issues.apache.org/jira/browse/CHUKWA-477
> Project: Hadoop Chukwa
> Issue Type: New Feature
> Components: Data Processors
> Reporter: Bill Graham
> Assignee: Bill Graham
> Attachments: CHUKWA-477.1.patch, CHUKWA-477.2.patch
>
>
> Add the ability to trigger an action upon successful completion of a Demux process.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-477) Support post-demux triggers
Posted by "Bill Graham (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bill Graham updated CHUKWA-477:
-------------------------------
Attachment: CHUKWA-477.3.patch
One last minor rev of this patch. The TriggerAction signature should take the more generic Configuration, not ChukwaConfiguration.
> Support post-demux triggers
> ---------------------------
>
> Key: CHUKWA-477
> URL: https://issues.apache.org/jira/browse/CHUKWA-477
> Project: Hadoop Chukwa
> Issue Type: New Feature
> Components: Data Processors
> Reporter: Bill Graham
> Assignee: Bill Graham
> Attachments: CHUKWA-477.1.patch, CHUKWA-477.2.patch, CHUKWA-477.3.patch
>
>
> Add the ability to trigger an action upon successful completion of a Demux process.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-477) Support post-demux triggers
Posted by "Bill Graham (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bill Graham updated CHUKWA-477:
-------------------------------
Attachment: CHUKWA-477.4.patch
Attached is CHUKWA-477.4.patch, which adds the concept of a TriggerEvent enum, which gets passed to the TriggerAction. Unless there are other comments, I'm code-compete on the trigger work.
I've been running my HttpTriggerAction (as discussed on the mailing list) with this patch and will submit that as a separate JIRA.
> Support post-demux triggers
> ---------------------------
>
> Key: CHUKWA-477
> URL: https://issues.apache.org/jira/browse/CHUKWA-477
> Project: Hadoop Chukwa
> Issue Type: New Feature
> Components: Data Processors
> Reporter: Bill Graham
> Assignee: Bill Graham
> Attachments: CHUKWA-477.1.patch, CHUKWA-477.2.patch, CHUKWA-477.3.patch, CHUKWA-477.4.patch
>
>
> Add the ability to trigger an action upon successful completion of a Demux process.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-477) Support post-demux triggers
Posted by "Bill Graham (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bill Graham updated CHUKWA-477:
-------------------------------
Attachment: CHUKWA-477.1.patch
Here's a first pass at this functionality (CHUKWA-477.1.patch) for review. Please review and provide feedback re design and naming. A few things to note:
- I've added the following constant and interface/package:
{noformat}
package org.apache.hadoop.chukwa.datatrigger;
public interface TriggerAction {
public void load(ChukwaConfiguration conf, FileSystem fs, FileStatus[] src) throws IOException;
}
public interface CHUKWA_CONSTANT {
public static final String POST_DEMUX_SUCCESS_ACTION = "chukwa.post.demux.success.action";
}
{noformat}
- The idea is that you can implement specific TriggerActions for to occur in different scenarios. They know how to react to some action, which is relayed via the FileStatus[] objects. The method signature of TriggerAction is the same as that of DataLoaderFactory, so the latter could implement the former if we want to standardize and go that route. Or we could create a new signature for TriggerAction that contains more context re the trigger itself.
- PostProcessorMananger does not know the names of the files that were added to repos/, since those details are never returned by MoveToRepository. I'd like to change MoveToRepository to return a collection of Paths to the action can see what's been added. This could be done in a separate JIRA if that makes more sense.
> Support post-demux triggers
> ---------------------------
>
> Key: CHUKWA-477
> URL: https://issues.apache.org/jira/browse/CHUKWA-477
> Project: Hadoop Chukwa
> Issue Type: New Feature
> Components: Data Processors
> Reporter: Bill Graham
> Assignee: Bill Graham
> Attachments: CHUKWA-477.1.patch
>
>
> Add the ability to trigger an action upon successful completion of a Demux process.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-477) Support post-demux triggers
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12858734#action_12858734 ]
Eric Yang commented on CHUKWA-477:
----------------------------------
+1 looks good.
> Support post-demux triggers
> ---------------------------
>
> Key: CHUKWA-477
> URL: https://issues.apache.org/jira/browse/CHUKWA-477
> Project: Hadoop Chukwa
> Issue Type: New Feature
> Components: Data Processors
> Reporter: Bill Graham
> Assignee: Bill Graham
> Attachments: CHUKWA-477.1.patch
>
>
> Add the ability to trigger an action upon successful completion of a Demux process.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-477) Support post-demux triggers
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Yang updated CHUKWA-477:
-----------------------------
Status: Resolved (was: Patch Available)
Resolution: Fixed
I committed this yesterday, thanks Bill.
> Support post-demux triggers
> ---------------------------
>
> Key: CHUKWA-477
> URL: https://issues.apache.org/jira/browse/CHUKWA-477
> Project: Hadoop Chukwa
> Issue Type: New Feature
> Components: Data Processors
> Reporter: Bill Graham
> Assignee: Bill Graham
> Attachments: CHUKWA-477.1.patch, CHUKWA-477.2.patch, CHUKWA-477.3.patch, CHUKWA-477.4.patch
>
>
> Add the ability to trigger an action upon successful completion of a Demux process.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.