You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@chukwa.apache.org by 唐 颖 <iv...@gmail.com> on 2013/07/08 15:54:09 UTC
About CHUKWA-581,Demux do not support user custom reducer package name.
We can write our own mapper ,because MapProcessorFactory uses
Class.forName(parserClass).getConstructor().newInstance();
since we config the parserClass in config file.
But the ReduceProcessorFactory inits the reducer class using the classname "org.apache.hadoop.chukwa.extraction.demux.processor.reducer."+ reduceType.
So the reducer class must be put in package org.apache.hadoop.chukwa.extraction.demux.processor.reducer and named reduceType.
Why not we put the reducer class name in config file either?
For exam, the {customdatatype_mapper} means mapper class ,{customdatatype_reducer} means reducer class.
This can give chukka more flexibility.
--
Best regards,
Ivy Tang
Re: About CHUKWA-581,Demux do not support user custom reducer package name.
Posted by Eric Yang <er...@gmail.com>.
Yes, Chunk.getDataType() is the right way to do it. Thanks
regards,
Eric
On Fri, Jul 12, 2013 at 2:01 AM, IvyTang <iv...@gmail.com> wrote:
> Fix:
> Don't need to add the argument dataType in
> AbstractProcessor.parse(String recordEntry,
> OutputCollector<ChukwaRecordKey, ChukwaRecord> output, Reporter reporter).
> I can get the datatype from the chunk.
> --
> Best regards,
>
> Ivy Tang
>
> On 2013-7-12, at 下午3:04, IvyTang <iv...@gmail.com> wrote:
>
> > Hi,Eric
> > In order to applying to all multiple datatypes in Demux,init the Reducer
> class depends on the ChukwaRecordKey's datatype . I need the datatype in
> ChukwaRecordKey.
> >
> > The ChukwaRecordKey needs to add a field named dataType , showing this
> ChukwaRecord's datatype . And also i need add an argument in
> > parse(String recordEntry, OutputCollector<ChukwaRecordKey, ChukwaRecord>
> output, Reporter reporter)
> >
> > An argument String dataType needs to be passed to this method to build
> the ChukwaRecordKey.
> >
> > Adding a field in ChukwaRecordKey ,and add an argument in method
> AbstractProcessor.parse(String recordEntry,
> OutputCollector<ChukwaRecordKey, ChukwaRecord> output, Reporter reporter)
> >
> > Is that acceptable? This is the only way i can think of to make the
> custom reducer more configurable.
> >
> > --
> > Best regards,
> >
> > Ivy Tang
> >
> > On 2013-7-10, at 下午12:01, Eric Yang <er...@gmail.com> wrote:
> >
> >> Yes that will help. Please include test case as well. Thanks Ivy.
> >>
> >> regards,
> >> Eric
> >>
> >>
> >> On Mon, Jul 8, 2013 at 10:42 PM, IvyTang <iv...@gmail.com> wrote:
> >>
> >>> This issue has been reported
> >>> https://issues.apache.org/jira/browse/CHUKWA-581.
> >>>
> >>> And i think i can submit a patch later .
> >>>
> >>> Should i open a new jira or just assign this CHUKWA-581 to myself?
> >>>
> >>> This can implement reducer outside chukwa.
> >>>
> >>> --
> >>> Best regards,
> >>>
> >>> Ivy Tang
> >>>
> >>> On 2013-7-9, at 下午1:22, Eric Yang <er...@gmail.com> wrote:
> >>>
> >>>> The recommended change can improve usability to implement parsers
> outside
> >>>> of Chukwa packages. +1 on the change. Do you mind open a jira for
> this?
> >>>> Thanks
> >>>>
> >>>> regards,
> >>>> Eric
> >>>>
> >>>>
> >>>> On Mon, Jul 8, 2013 at 6:54 AM, 唐 颖 <iv...@gmail.com> wrote:
> >>>>
> >>>>> We can write our own mapper ,because MapProcessorFactory uses
> >>>>> Class.forName(parserClass).getConstructor().newInstance();
> >>>>> since we config the parserClass in config file.
> >>>>>
> >>>>> But the ReduceProcessorFactory inits the reducer class using the
> >>>>> classname
> >>> "org.apache.hadoop.chukwa.extraction.demux.processor.reducer."+
> >>>>> reduceType.
> >>>>> So the reducer class must be put in package
> >>>>> org.apache.hadoop.chukwa.extraction.demux.processor.reducer and named
> >>>>> reduceType.
> >>>>>
> >>>>> Why not we put the reducer class name in config file either?
> >>>>> For exam, the {customdatatype_mapper} means mapper class
> >>>>> ,{customdatatype_reducer} means reducer class.
> >>>>>
> >>>>> This can give chukka more flexibility.
> >>>>>
> >>>>> --
> >>>>> Best regards,
> >>>>>
> >>>>> Ivy Tang
> >>>>>
> >>>>>
> >>>
> >>>
> >
>
>
Re: About CHUKWA-581,Demux do not support user custom reducer package name.
Posted by IvyTang <iv...@gmail.com>.
Fix:
Don't need to add the argument dataType in AbstractProcessor.parse(String recordEntry, OutputCollector<ChukwaRecordKey, ChukwaRecord> output, Reporter reporter).
I can get the datatype from the chunk.
--
Best regards,
Ivy Tang
On 2013-7-12, at 下午3:04, IvyTang <iv...@gmail.com> wrote:
> Hi,Eric
> In order to applying to all multiple datatypes in Demux,init the Reducer class depends on the ChukwaRecordKey's datatype . I need the datatype in ChukwaRecordKey.
>
> The ChukwaRecordKey needs to add a field named dataType , showing this ChukwaRecord's datatype . And also i need add an argument in
> parse(String recordEntry, OutputCollector<ChukwaRecordKey, ChukwaRecord> output, Reporter reporter)
>
> An argument String dataType needs to be passed to this method to build the ChukwaRecordKey.
>
> Adding a field in ChukwaRecordKey ,and add an argument in method AbstractProcessor.parse(String recordEntry, OutputCollector<ChukwaRecordKey, ChukwaRecord> output, Reporter reporter)
>
> Is that acceptable? This is the only way i can think of to make the custom reducer more configurable.
>
> --
> Best regards,
>
> Ivy Tang
>
> On 2013-7-10, at 下午12:01, Eric Yang <er...@gmail.com> wrote:
>
>> Yes that will help. Please include test case as well. Thanks Ivy.
>>
>> regards,
>> Eric
>>
>>
>> On Mon, Jul 8, 2013 at 10:42 PM, IvyTang <iv...@gmail.com> wrote:
>>
>>> This issue has been reported
>>> https://issues.apache.org/jira/browse/CHUKWA-581.
>>>
>>> And i think i can submit a patch later .
>>>
>>> Should i open a new jira or just assign this CHUKWA-581 to myself?
>>>
>>> This can implement reducer outside chukwa.
>>>
>>> --
>>> Best regards,
>>>
>>> Ivy Tang
>>>
>>> On 2013-7-9, at 下午1:22, Eric Yang <er...@gmail.com> wrote:
>>>
>>>> The recommended change can improve usability to implement parsers outside
>>>> of Chukwa packages. +1 on the change. Do you mind open a jira for this?
>>>> Thanks
>>>>
>>>> regards,
>>>> Eric
>>>>
>>>>
>>>> On Mon, Jul 8, 2013 at 6:54 AM, 唐 颖 <iv...@gmail.com> wrote:
>>>>
>>>>> We can write our own mapper ,because MapProcessorFactory uses
>>>>> Class.forName(parserClass).getConstructor().newInstance();
>>>>> since we config the parserClass in config file.
>>>>>
>>>>> But the ReduceProcessorFactory inits the reducer class using the
>>>>> classname
>>> "org.apache.hadoop.chukwa.extraction.demux.processor.reducer."+
>>>>> reduceType.
>>>>> So the reducer class must be put in package
>>>>> org.apache.hadoop.chukwa.extraction.demux.processor.reducer and named
>>>>> reduceType.
>>>>>
>>>>> Why not we put the reducer class name in config file either?
>>>>> For exam, the {customdatatype_mapper} means mapper class
>>>>> ,{customdatatype_reducer} means reducer class.
>>>>>
>>>>> This can give chukka more flexibility.
>>>>>
>>>>> --
>>>>> Best regards,
>>>>>
>>>>> Ivy Tang
>>>>>
>>>>>
>>>
>>>
>
Re: About CHUKWA-581,Demux do not support user custom reducer package name.
Posted by IvyTang <iv...@gmail.com>.
Hi,Eric
In order to applying to all multiple datatypes in Demux,init the Reducer class depends on the ChukwaRecordKey's datatype . I need the datatype in ChukwaRecordKey.
The ChukwaRecordKey needs to add a field named dataType , showing this ChukwaRecord's datatype . And also i need add an argument in
parse(String recordEntry, OutputCollector<ChukwaRecordKey, ChukwaRecord> output, Reporter reporter)
An argument String dataType needs to be passed to this method to build the ChukwaRecordKey.
Adding a field in ChukwaRecordKey ,and add an argument in method AbstractProcessor.parse(String recordEntry, OutputCollector<ChukwaRecordKey, ChukwaRecord> output, Reporter reporter)
Is that acceptable? This is the only way i can think of to make the custom reducer more configurable.
--
Best regards,
Ivy Tang
On 2013-7-10, at 下午12:01, Eric Yang <er...@gmail.com> wrote:
> Yes that will help. Please include test case as well. Thanks Ivy.
>
> regards,
> Eric
>
>
> On Mon, Jul 8, 2013 at 10:42 PM, IvyTang <iv...@gmail.com> wrote:
>
>> This issue has been reported
>> https://issues.apache.org/jira/browse/CHUKWA-581.
>>
>> And i think i can submit a patch later .
>>
>> Should i open a new jira or just assign this CHUKWA-581 to myself?
>>
>> This can implement reducer outside chukwa.
>>
>> --
>> Best regards,
>>
>> Ivy Tang
>>
>> On 2013-7-9, at 下午1:22, Eric Yang <er...@gmail.com> wrote:
>>
>>> The recommended change can improve usability to implement parsers outside
>>> of Chukwa packages. +1 on the change. Do you mind open a jira for this?
>>> Thanks
>>>
>>> regards,
>>> Eric
>>>
>>>
>>> On Mon, Jul 8, 2013 at 6:54 AM, 唐 颖 <iv...@gmail.com> wrote:
>>>
>>>> We can write our own mapper ,because MapProcessorFactory uses
>>>> Class.forName(parserClass).getConstructor().newInstance();
>>>> since we config the parserClass in config file.
>>>>
>>>> But the ReduceProcessorFactory inits the reducer class using the
>>>> classname
>> "org.apache.hadoop.chukwa.extraction.demux.processor.reducer."+
>>>> reduceType.
>>>> So the reducer class must be put in package
>>>> org.apache.hadoop.chukwa.extraction.demux.processor.reducer and named
>>>> reduceType.
>>>>
>>>> Why not we put the reducer class name in config file either?
>>>> For exam, the {customdatatype_mapper} means mapper class
>>>> ,{customdatatype_reducer} means reducer class.
>>>>
>>>> This can give chukka more flexibility.
>>>>
>>>> --
>>>> Best regards,
>>>>
>>>> Ivy Tang
>>>>
>>>>
>>
>>
Re: About CHUKWA-581,Demux do not support user custom reducer package name.
Posted by Eric Yang <er...@gmail.com>.
Yes that will help. Please include test case as well. Thanks Ivy.
regards,
Eric
On Mon, Jul 8, 2013 at 10:42 PM, IvyTang <iv...@gmail.com> wrote:
> This issue has been reported
> https://issues.apache.org/jira/browse/CHUKWA-581.
>
> And i think i can submit a patch later .
>
> Should i open a new jira or just assign this CHUKWA-581 to myself?
>
> This can implement reducer outside chukwa.
>
> --
> Best regards,
>
> Ivy Tang
>
> On 2013-7-9, at 下午1:22, Eric Yang <er...@gmail.com> wrote:
>
> > The recommended change can improve usability to implement parsers outside
> > of Chukwa packages. +1 on the change. Do you mind open a jira for this?
> > Thanks
> >
> > regards,
> > Eric
> >
> >
> > On Mon, Jul 8, 2013 at 6:54 AM, 唐 颖 <iv...@gmail.com> wrote:
> >
> >> We can write our own mapper ,because MapProcessorFactory uses
> >> Class.forName(parserClass).getConstructor().newInstance();
> >> since we config the parserClass in config file.
> >>
> >> But the ReduceProcessorFactory inits the reducer class using the
> >> classname
> "org.apache.hadoop.chukwa.extraction.demux.processor.reducer."+
> >> reduceType.
> >> So the reducer class must be put in package
> >> org.apache.hadoop.chukwa.extraction.demux.processor.reducer and named
> >> reduceType.
> >>
> >> Why not we put the reducer class name in config file either?
> >> For exam, the {customdatatype_mapper} means mapper class
> >> ,{customdatatype_reducer} means reducer class.
> >>
> >> This can give chukka more flexibility.
> >>
> >> --
> >> Best regards,
> >>
> >> Ivy Tang
> >>
> >>
>
>
Re: About CHUKWA-581,Demux do not support user custom reducer package name.
Posted by IvyTang <iv...@gmail.com>.
This issue has been reported https://issues.apache.org/jira/browse/CHUKWA-581.
And i think i can submit a patch later .
Should i open a new jira or just assign this CHUKWA-581 to myself?
This can implement reducer outside chukwa.
--
Best regards,
Ivy Tang
On 2013-7-9, at 下午1:22, Eric Yang <er...@gmail.com> wrote:
> The recommended change can improve usability to implement parsers outside
> of Chukwa packages. +1 on the change. Do you mind open a jira for this?
> Thanks
>
> regards,
> Eric
>
>
> On Mon, Jul 8, 2013 at 6:54 AM, 唐 颖 <iv...@gmail.com> wrote:
>
>> We can write our own mapper ,because MapProcessorFactory uses
>> Class.forName(parserClass).getConstructor().newInstance();
>> since we config the parserClass in config file.
>>
>> But the ReduceProcessorFactory inits the reducer class using the
>> classname "org.apache.hadoop.chukwa.extraction.demux.processor.reducer."+
>> reduceType.
>> So the reducer class must be put in package
>> org.apache.hadoop.chukwa.extraction.demux.processor.reducer and named
>> reduceType.
>>
>> Why not we put the reducer class name in config file either?
>> For exam, the {customdatatype_mapper} means mapper class
>> ,{customdatatype_reducer} means reducer class.
>>
>> This can give chukka more flexibility.
>>
>> --
>> Best regards,
>>
>> Ivy Tang
>>
>>
Re: About CHUKWA-581,Demux do not support user custom reducer package name.
Posted by Eric Yang <er...@gmail.com>.
The recommended change can improve usability to implement parsers outside
of Chukwa packages. +1 on the change. Do you mind open a jira for this?
Thanks
regards,
Eric
On Mon, Jul 8, 2013 at 6:54 AM, 唐 颖 <iv...@gmail.com> wrote:
> We can write our own mapper ,because MapProcessorFactory uses
> Class.forName(parserClass).getConstructor().newInstance();
> since we config the parserClass in config file.
>
> But the ReduceProcessorFactory inits the reducer class using the
> classname "org.apache.hadoop.chukwa.extraction.demux.processor.reducer."+
> reduceType.
> So the reducer class must be put in package
> org.apache.hadoop.chukwa.extraction.demux.processor.reducer and named
> reduceType.
>
> Why not we put the reducer class name in config file either?
> For exam, the {customdatatype_mapper} means mapper class
> ,{customdatatype_reducer} means reducer class.
>
> This can give chukka more flexibility.
>
> --
> Best regards,
>
> Ivy Tang
>
>