You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@chukwa.apache.org by Bill Graham <bi...@gmail.com> on 2010/04/22 00:31:41 UTC
HttpTriggerAction - configuring N objects
Hi,
As a follow up to CHUKWA-477, I'm writing a class called HttpTriggerAction
that can hit one or more URLs upon successful completion of a demux job. I'd
like to contribute it back unless anyone objects. Anyway, I'm looking for
feedback though on how to configure this object.
The issue is that since the class can hit N urls, it needs N sets of
key-value configurations. The hadoop configurations model is just name-value
pairs though, so I'm kicking around ideas around the best way to handle
this.
Specifically, I need to configure values for url, an optional HTTP method
(default is GET), an optional collection of HTTP headers and an optional
post body. I was thinking of just making a convention where key values could
be incremented like below, but wanted to see if there were better
suggestions out there.
chukwa.trigger.action.[eventName].http.1.url=http://site.com/firstTrigger
chukwa.trigger.action.[eventName].http.1.headers=User-Agent:chukwa
chukwa.trigger.action.[eventName].http.2.url=http://site.com/secondTrigger
chukwa.trigger.action.[eventName].http.2.method=POST
chukwa.trigger.action.[eventName].http.2.headers=User-Agent:chukwa,Accepts:text/plain
chukwa.trigger.action.[eventName].http.2.body=Some post body to submit
....
chukwa.trigger.action.[eventName].http.N.url=
chukwa.trigger.action.[eventName].http.N.method=
chukwa.trigger.action.[eventName].http.N.headers=
chukwa.trigger.action.[eventName].http.N.body=
Since the action could potentially be used by other types of events, the
event name should be included. This implies that we should add an eventName
field to the TriggerAction.execute method in CHUKWA-477.
Thoughts?
thanks,
Bill
Re: HttpTriggerAction - configuring N objects
Posted by Bill Graham <bi...@gmail.com>.
Thanks guys. Eric, I think I'm on a similar path to what you're suggesting
except I'm using a two-step config to specidy a.) the action classes, and
b.) their configs. Jerome, I considered that, but gets tricky with
multi-value sets (like headers). At some point we get into delimiter
overload.
To get to specifics, here's an example of what I currently have:
- A comma separated list of TriggerAction classes to be invoked upon
successful completion of a successful demux run. This is similar to the
current data loader pattern and can also be used wherever else we later want
to fire TriggerActions.
<property>
<name>chukwa.post.demux.success.action</name>
<value>org.apache.hadoop.chukwa.extraction.demux.HttpTriggerAction,some.other.TriggerAction</value>
</property>
- Then when the HttpTriggerAction runs, it knows how to look for it's
configs in the way that we've been discussing. For the above action, the
trigger event name will be passed to TriggerAction as an enum. The enum
would have a name like 'postDemuxSuccess', as well as a pointer to the base
config string for the event, like 'chukwa.trigger.post.demux.success'.
- HttpTriggerAction would in this case then know to look for it's configs
under
chukwa.trigger.post.demux.success.*http *and look for values like this:
<property>
<name>chukwa.trigger.post.demux.success.http.1.url</name>
<value>http://site.com/firstTrigger</value>
</property>
so the syntax of the config key then is:
chukwa.[eventName].[actionNS].N.[actionKeys]
Although other actions are free to implement all parts below eventName
however makes the most sense for their needs.
thanks,
Bill
On Wed, Apr 21, 2010 at 4:50 PM, Jerome Boulon <jb...@netflix.com> wrote:
> Hi,
> You can have a root configuration key that will give the list of keys to
> look for:
> Or you can look for the existence of http.N key in the configuration
> object.
>
> Ex
> <property >
> <name>myConfig.eventName.list</name>
> <value>http.1, http.2, ..., http.x</value>
> </property>
>
>
> Can you clarify what the eventName is?
>
> /Jerome.
>
>
> On 4/21/10 3:31 PM, "Bill Graham" <bi...@gmail.com> wrote:
>
> Hi,
>
> As a follow up to CHUKWA-477, I'm writing a class called HttpTriggerAction
> that can hit one or more URLs upon successful completion of a demux job. I'd
> like to contribute it back unless anyone objects. Anyway, I'm looking for
> feedback though on how to configure this object.
>
> The issue is that since the class can hit N urls, it needs N sets of
> key-value configurations. The hadoop configurations model is just name-value
> pairs though, so I'm kicking around ideas around the best way to handle
> this.
>
> Specifically, I need to configure values for url, an optional HTTP method
> (default is GET), an optional collection of HTTP headers and an optional
> post body. I was thinking of just making a convention where key values could
> be incremented like below, but wanted to see if there were better
> suggestions out there.
>
> chukwa.trigger.action.[eventName].http.1.url=http://site.com/firstTrigger
> chukwa.trigger.action.[eventName].http.1.headers=User-Agent:chukwa
>
> chukwa.trigger.action.[eventName].http.2.url=http://site.com/secondTrigger
> chukwa.trigger.action.[eventName].http.2.method=POST
>
> chukwa.trigger.action.[eventName].http.2.headers=User-Agent:chukwa,Accepts:text/plain
> chukwa.trigger.action.[eventName].http.2.body=Some post body to submit
> ....
> chukwa.trigger.action.[eventName].http.N.url=
> chukwa.trigger.action.[eventName].http.N.method=
> chukwa.trigger.action.[eventName].http.N.headers=
> chukwa.trigger.action.[eventName].http.N.body=
>
> Since the action could potentially be used by other types of events, the
> event name should be included. This implies that we should add an eventName
> field to the TriggerAction.execute method in CHUKWA-477.
>
> Thoughts?
>
> thanks,
> Bill
>
>
>
>
Re: HttpTriggerAction - configuring N objects
Posted by Jerome Boulon <jb...@netflix.com>.
Hi,
You can have a root configuration key that will give the list of keys to
look for:
Or you can look for the existence of http.N key in the configuration object.
Ex
<property >
<name>myConfig.eventName.list</name>
<value>http.1, http.2, ..., http.x</value>
</property>
Can you clarify what the eventName is?
/Jerome.
On 4/21/10 3:31 PM, "Bill Graham" <bi...@gmail.com> wrote:
> Hi,
>
> As a follow up to CHUKWA-477, I'm writing a class called HttpTriggerAction
> that can hit one or more URLs upon successful completion of a demux job. I'd
> like to contribute it back unless anyone objects. Anyway, I'm looking for
> feedback though on how to configure this object.
>
> The issue is that since the class can hit N urls, it needs N sets of key-value
> configurations. The hadoop configurations model is just name-value pairs
> though, so I'm kicking around ideas around the best way to handle this.
>
> Specifically, I need to configure values for url, an optional HTTP method
> (default is GET), an optional collection of HTTP headers and an optional post
> body. I was thinking of just making a convention where key values could be
> incremented like below, but wanted to see if there were better suggestions out
> there.
>
> chukwa.trigger.action.[eventName].http.1.url=http://site.com/firstTrigger
> chukwa.trigger.action.[eventName].http.1.headers=User-Agent:chukwa
>
> chukwa.trigger.action.[eventName].http.2.url=http://site.com/secondTrigger
> chukwa.trigger.action.[eventName].http.2.method=POST
> chukwa.trigger.action.[eventName].http.2.headers=User-Agent:chukwa,Accepts:tex
> t/plain
> chukwa.trigger.action.[eventName].http.2.body=Some post body to submit
> ....
> chukwa.trigger.action.[eventName].http.N.url=
> chukwa.trigger.action.[eventName].http.N.method=
> chukwa.trigger.action.[eventName].http.N.headers=
> chukwa.trigger.action.[eventName].http.N.body=
>
> Since the action could potentially be used by other types of events, the event
> name should be included. This implies that we should add an eventName field to
> the TriggerAction.execute method in CHUKWA-477.
>
> Thoughts?
>
> thanks,
> Bill
>
>
>
Re: HttpTriggerAction - configuring N objects
Posted by Eric Yang <ey...@yahoo-inc.com>.
I don¹t have good solution to simplify multi-dimension config parameters,
but for the keyword style, it might be useful to have:
Chukwa.trigger.action.HttpTriggerAction.[eventName].1.url=
Chukwa.trigger.action.HttpTriggerAction.[eventName].1.headers=
Hence, we know which Java class to instantiate, and potentially pass in
[eventName], index, type into a configure method.
It could be written like this in hadoop style configuration file:
<property>
<name>Chukwa.trigger.action.HttpTriggerAction.[eventName].1.url</name>
<value>http://site.com/firstTrigger</value>
</property>
<property>
<name>Chukwa.trigger.action.HttpTriggerAction.[eventName].1.headers</name>
<value>User-Agent:chukwa</value>
</property>
It¹s a possible method to automate config generation using Configuration
object from hadoop and enable a path to build GUI based config editor in
HICC.
Regards,
Eric
On 4/21/10 3:31 PM, "Bill Graham" <bi...@gmail.com> wrote:
> Hi,
>
> As a follow up to CHUKWA-477, I'm writing a class called HttpTriggerAction
> that can hit one or more URLs upon successful completion of a demux job. I'd
> like to contribute it back unless anyone objects. Anyway, I'm looking for
> feedback though on how to configure this object.
>
> The issue is that since the class can hit N urls, it needs N sets of key-value
> configurations. The hadoop configurations model is just name-value pairs
> though, so I'm kicking around ideas around the best way to handle this.
>
> Specifically, I need to configure values for url, an optional HTTP method
> (default is GET), an optional collection of HTTP headers and an optional post
> body. I was thinking of just making a convention where key values could be
> incremented like below, but wanted to see if there were better suggestions out
> there.
>
> chukwa.trigger.action.[eventName].http.1.url=http://site.com/firstTrigger
> chukwa.trigger.action.[eventName].http.1.headers=User-Agent:chukwa
>
> chukwa.trigger.action.[eventName].http.2.url=http://site.com/secondTrigger
> chukwa.trigger.action.[eventName].http.2.method=POST
> chukwa.trigger.action.[eventName].http.2.headers=User-Agent:chukwa,Accepts:tex
> t/plain
> chukwa.trigger.action.[eventName].http.2.body=Some post body to submit
> ....
> chukwa.trigger.action.[eventName].http.N.url=
> chukwa.trigger.action.[eventName].http.N.method=
> chukwa.trigger.action.[eventName].http.N.headers=
> chukwa.trigger.action.[eventName].http.N.body=
>
> Since the action could potentially be used by other types of events, the event
> name should be included. This implies that we should add an eventName field to
> the TriggerAction.execute method in CHUKWA-477.
>
> Thoughts?
>
> thanks,
> Bill
>
>
>