You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@chukwa.apache.org by "Yingzhong Xu (JIRA)" <ji...@apache.org> on 2011/02/23 03:48:38 UTC
[jira] Created: (CHUKWA-581) Demux do not support user custom
reducer package name.
Demux do not support user custom reducer package name.
-------------------------------------------------------
Key: CHUKWA-581
URL: https://issues.apache.org/jira/browse/CHUKWA-581
Project: Chukwa
Issue Type: Improvement
Components: MR Data Processors
Affects Versions: 0.4.0
Environment: Ubuntu 10.0.4,Hadoop 0.20
Reporter: Yingzhong Xu
Priority: Critical
Fix For: 0.4.0
In ReduceProcessorFactory.java,
it only gets {reduceType}.class from this package,as following:
bq. String path = "org.apache.hadoop.chukwa.extraction.demux.processor.reducer." + reduceType;
So our custom reducer could only be placed in that package.
Thus, When we add our {demux}.jar to Hadoop classpath, since the Hadoop framework will not be able to merge the class files from multiple jar files with the same package name, custom reduce class can not be found.
I get a way to overcome this defect.Put custom reducer into a package named with "org.apache.hadoop.chukwa.extraction.demux.processor.reducer."+{custom}
Here,{custom} is a part of package name.Then in parser,we can write like this:
bq. Key.setReducerType("{custom}.YourReducer")
In this way, Hadoop framework could be able to find our reduce class in this partial user defined new package.
But I think it will be better to fix chukwa to fully support custom package name for reducer.
Thanks for Eric Yang's help.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CHUKWA-581) Demux do not support user custom
reducer package name.
Posted by "Yingzhong Xu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yingzhong Xu updated CHUKWA-581:
--------------------------------
Description:
In ReduceProcessorFactory.java,
it only gets {reduceType}.class from this package,as following:
bq. String path = "org.apache.hadoop.chukwa.extraction.demux.processor.reducer." + reduceType;
So our custom reducer could only be placed in that package.
Thus, When we add our {demux}.jar to Hadoop classpath, since the Hadoop framework will not be able to merge the class files from multiple jar files with the same package name, custom reduce class can not be found.
I get a way to overcome this defect.Put custom reducer into a package named with "org.apache.hadoop.chukwa.extraction.demux.processor.reducer."+{custom}
Here,{custom} is a part of package name.Then in parser,we can write like this:
bq. Key.setReducerType("{custom}.YourReducer")
In this way, Hadoop framework could be able to find our reduce class in this partial user defined new package.
But I think it will be better to fix chukwa to remove the default package name and fully support custom package name for reducer.
Thanks for Eric Yang's help.
was:
In ReduceProcessorFactory.java,
it only gets {reduceType}.class from this package,as following:
bq. String path = "org.apache.hadoop.chukwa.extraction.demux.processor.reducer." + reduceType;
So our custom reducer could only be placed in that package.
Thus, When we add our {demux}.jar to Hadoop classpath, since the Hadoop framework will not be able to merge the class files from multiple jar files with the same package name, custom reduce class can not be found.
I get a way to overcome this defect.Put custom reducer into a package named with "org.apache.hadoop.chukwa.extraction.demux.processor.reducer."+{custom}
Here,{custom} is a part of package name.Then in parser,we can write like this:
bq. Key.setReducerType("{custom}.YourReducer")
In this way, Hadoop framework could be able to find our reduce class in this partial user defined new package.
But I think it will be better to fix chukwa to fully support custom package name for reducer.
Thanks for Eric Yang's help.
> Demux do not support user custom reducer package name.
> -------------------------------------------------------
>
> Key: CHUKWA-581
> URL: https://issues.apache.org/jira/browse/CHUKWA-581
> Project: Chukwa
> Issue Type: Improvement
> Components: MR Data Processors
> Affects Versions: 0.4.0
> Environment: Ubuntu 10.0.4,Hadoop 0.20
> Reporter: Yingzhong Xu
> Priority: Critical
> Fix For: 0.4.0
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> In ReduceProcessorFactory.java,
> it only gets {reduceType}.class from this package,as following:
> bq. String path = "org.apache.hadoop.chukwa.extraction.demux.processor.reducer." + reduceType;
> So our custom reducer could only be placed in that package.
> Thus, When we add our {demux}.jar to Hadoop classpath, since the Hadoop framework will not be able to merge the class files from multiple jar files with the same package name, custom reduce class can not be found.
> I get a way to overcome this defect.Put custom reducer into a package named with "org.apache.hadoop.chukwa.extraction.demux.processor.reducer."+{custom}
> Here,{custom} is a part of package name.Then in parser,we can write like this:
> bq. Key.setReducerType("{custom}.YourReducer")
> In this way, Hadoop framework could be able to find our reduce class in this partial user defined new package.
> But I think it will be better to fix chukwa to remove the default package name and fully support custom package name for reducer.
> Thanks for Eric Yang's help.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CHUKWA-581) Demux do not support user custom
reducer package name.
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Yang updated CHUKWA-581:
-----------------------------
Fix Version/s: (was: 0.4.0)
> Demux do not support user custom reducer package name.
> -------------------------------------------------------
>
> Key: CHUKWA-581
> URL: https://issues.apache.org/jira/browse/CHUKWA-581
> Project: Chukwa
> Issue Type: Improvement
> Components: MR Data Processors
> Affects Versions: 0.4.0
> Environment: Ubuntu 10.0.4,Hadoop 0.20
> Reporter: Yingzhong Xu
> Priority: Critical
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> In ReduceProcessorFactory.java,
> it only gets {reduceType}.class from this package,as following:
> bq. String path = "org.apache.hadoop.chukwa.extraction.demux.processor.reducer." + reduceType;
> So our custom reducer could only be placed in that package.
> Thus, When we add our {demux}.jar to Hadoop classpath, since the Hadoop framework will not be able to merge the class files from multiple jar files with the same package name, custom reduce class can not be found.
> I get a way to overcome this defect.Put custom reducer into a package named with "org.apache.hadoop.chukwa.extraction.demux.processor.reducer."+{custom}
> Here,{custom} is a part of package name.Then in parser,we can write like this:
> bq. Key.setReducerType("{custom}.YourReducer")
> In this way, Hadoop framework could be able to find our reduce class in this partial user defined new package.
> But I think it will be better to fix chukwa to remove the default package name and fully support custom package name for reducer.
> Thanks for Eric Yang's help.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira