You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@chukwa.apache.org by "Jerome Boulon (JIRA)" <ji...@apache.org> on 2009/06/02 19:45:07 UTC
[jira] Created: (CHUKWA-276) Hourly and Daily rolling should use a
single reducer
Hourly and Daily rolling should use a single reducer
----------------------------------------------------
Key: CHUKWA-276
URL: https://issues.apache.org/jira/browse/CHUKWA-276
Project: Hadoop Chukwa
Issue Type: Improvement
Reporter: Jerome Boulon
Hourly and Daily rolling should not use more than one reducer and they are currently relying on the default parameter to be correct.
<property>
<name>mapred.reduce.tasks</name>
<value>1</value>
<description>The default number of reduce tasks per job. Typically set
to a prime close to the number of available hosts. Ignored when
mapred.job.tracker is "local".
</description>
</property>
This should be forced within the Rolling class in order to avoid any issue.
And ideally, the code should be changed to allow more than one reducer.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-276) Hourly and Daily rolling should use
a single reducer + fix classpath issue
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12723113#action_12723113 ]
Hudson commented on CHUKWA-276:
-------------------------------
Integrated in Chukwa-trunk #61 (See [http://hudson.zones.apache.org/hudson/job/Chukwa-trunk/61/])
> Hourly and Daily rolling should use a single reducer + fix classpath issue
> --------------------------------------------------------------------------
>
> Key: CHUKWA-276
> URL: https://issues.apache.org/jira/browse/CHUKWA-276
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: Data Processors
> Reporter: Jerome Boulon
> Assignee: Jerome Boulon
> Fix For: 0.2.0, 0.3.0
>
> Attachments: CHUKWA-276.patch
>
>
> Hourly and Daily rolling should not use more than one reducer and they are currently relying on the default parameter to be correct.
> <property>
> <name>mapred.reduce.tasks</name>
> <value>1</value>
> <description>The default number of reduce tasks per job. Typically set
> to a prime close to the number of available hosts. Ignored when
> mapred.job.tracker is "local".
> </description>
> </property>
> This should be forced within the Rolling class in order to avoid any issue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-276) Hourly and Daily rolling should use a
single reducer
Posted by "Jerome Boulon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jerome Boulon updated CHUKWA-276:
---------------------------------
Component/s: Data Processors
Assignee: Jerome Boulon
> Hourly and Daily rolling should use a single reducer
> ----------------------------------------------------
>
> Key: CHUKWA-276
> URL: https://issues.apache.org/jira/browse/CHUKWA-276
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: Data Processors
> Reporter: Jerome Boulon
> Assignee: Jerome Boulon
>
> Hourly and Daily rolling should not use more than one reducer and they are currently relying on the default parameter to be correct.
> <property>
> <name>mapred.reduce.tasks</name>
> <value>1</value>
> <description>The default number of reduce tasks per job. Typically set
> to a prime close to the number of available hosts. Ignored when
> mapred.job.tracker is "local".
> </description>
> </property>
> This should be forced within the Rolling class in order to avoid any issue.
> And ideally, the code should be changed to allow more than one reducer.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-276) Hourly and Daily rolling should use a
single reducer + fix classpath issue
Posted by "Jerome Boulon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jerome Boulon updated CHUKWA-276:
---------------------------------
Description:
Hourly and Daily rolling should not use more than one reducer and they are currently relying on the default parameter to be correct.
<property>
<name>mapred.reduce.tasks</name>
<value>1</value>
<description>The default number of reduce tasks per job. Typically set
to a prime close to the number of available hosts. Ignored when
mapred.job.tracker is "local".
</description>
</property>
This should be forced within the Rolling class in order to avoid any issue.
was:
Hourly and Daily rolling should not use more than one reducer and they are currently relying on the default parameter to be correct.
<property>
<name>mapred.reduce.tasks</name>
<value>1</value>
<description>The default number of reduce tasks per job. Typically set
to a prime close to the number of available hosts. Ignored when
mapred.job.tracker is "local".
</description>
</property>
This should be forced within the Rolling class in order to avoid any issue.
And ideally, the code should be changed to allow more than one reducer.
Summary: Hourly and Daily rolling should use a single reducer + fix classpath issue (was: Hourly and Daily rolling should use a single reducer)
> Hourly and Daily rolling should use a single reducer + fix classpath issue
> --------------------------------------------------------------------------
>
> Key: CHUKWA-276
> URL: https://issues.apache.org/jira/browse/CHUKWA-276
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: Data Processors
> Reporter: Jerome Boulon
> Assignee: Jerome Boulon
>
> Hourly and Daily rolling should not use more than one reducer and they are currently relying on the default parameter to be correct.
> <property>
> <name>mapred.reduce.tasks</name>
> <value>1</value>
> <description>The default number of reduce tasks per job. Typically set
> to a prime close to the number of available hosts. Ignored when
> mapred.job.tracker is "local".
> </description>
> </property>
> This should be forced within the Rolling class in order to avoid any issue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-276) Hourly and Daily rolling should use a
single reducer + fix classpath issue
Posted by "Jerome Boulon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jerome Boulon updated CHUKWA-276:
---------------------------------
Fix Version/s: 0.3.0
0.2.0
Status: Patch Available (was: Open)
> Hourly and Daily rolling should use a single reducer + fix classpath issue
> --------------------------------------------------------------------------
>
> Key: CHUKWA-276
> URL: https://issues.apache.org/jira/browse/CHUKWA-276
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: Data Processors
> Reporter: Jerome Boulon
> Assignee: Jerome Boulon
> Fix For: 0.2.0, 0.3.0
>
> Attachments: CHUKWA-276.patch
>
>
> Hourly and Daily rolling should not use more than one reducer and they are currently relying on the default parameter to be correct.
> <property>
> <name>mapred.reduce.tasks</name>
> <value>1</value>
> <description>The default number of reduce tasks per job. Typically set
> to a prime close to the number of available hosts. Ignored when
> mapred.job.tracker is "local".
> </description>
> </property>
> This should be forced within the Rolling class in order to avoid any issue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-276) Hourly and Daily rolling should use a
single reducer + fix classpath issue
Posted by "Jerome Boulon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jerome Boulon updated CHUKWA-276:
---------------------------------
Attachment: CHUKWA-276.patch
- Fix classpath: add HADOOP_CONF_DIR
- Force reducer count to 1
> Hourly and Daily rolling should use a single reducer + fix classpath issue
> --------------------------------------------------------------------------
>
> Key: CHUKWA-276
> URL: https://issues.apache.org/jira/browse/CHUKWA-276
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: Data Processors
> Reporter: Jerome Boulon
> Assignee: Jerome Boulon
> Fix For: 0.2.0, 0.3.0
>
> Attachments: CHUKWA-276.patch
>
>
> Hourly and Daily rolling should not use more than one reducer and they are currently relying on the default parameter to be correct.
> <property>
> <name>mapred.reduce.tasks</name>
> <value>1</value>
> <description>The default number of reduce tasks per job. Typically set
> to a prime close to the number of available hosts. Ignored when
> mapred.job.tracker is "local".
> </description>
> </property>
> This should be forced within the Rolling class in order to avoid any issue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-276) Hourly and Daily rolling should use
a single reducer + fix classpath issue
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12722036#action_12722036 ]
Eric Yang commented on CHUKWA-276:
----------------------------------
+1 looks good.
> Hourly and Daily rolling should use a single reducer + fix classpath issue
> --------------------------------------------------------------------------
>
> Key: CHUKWA-276
> URL: https://issues.apache.org/jira/browse/CHUKWA-276
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: Data Processors
> Reporter: Jerome Boulon
> Assignee: Jerome Boulon
> Fix For: 0.2.0, 0.3.0
>
> Attachments: CHUKWA-276.patch
>
>
> Hourly and Daily rolling should not use more than one reducer and they are currently relying on the default parameter to be correct.
> <property>
> <name>mapred.reduce.tasks</name>
> <value>1</value>
> <description>The default number of reduce tasks per job. Typically set
> to a prime close to the number of available hosts. Ignored when
> mapred.job.tracker is "local".
> </description>
> </property>
> This should be forced within the Rolling class in order to avoid any issue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-276) Hourly and Daily rolling should use a
single reducer + fix classpath issue
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Yang updated CHUKWA-276:
-----------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this to trunk and 0.2, thanks Jerome.
> Hourly and Daily rolling should use a single reducer + fix classpath issue
> --------------------------------------------------------------------------
>
> Key: CHUKWA-276
> URL: https://issues.apache.org/jira/browse/CHUKWA-276
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: Data Processors
> Reporter: Jerome Boulon
> Assignee: Jerome Boulon
> Fix For: 0.2.0, 0.3.0
>
> Attachments: CHUKWA-276.patch
>
>
> Hourly and Daily rolling should not use more than one reducer and they are currently relying on the default parameter to be correct.
> <property>
> <name>mapred.reduce.tasks</name>
> <value>1</value>
> <description>The default number of reduce tasks per job. Typically set
> to a prime close to the number of available hosts. Ignored when
> mapred.job.tracker is "local".
> </description>
> </property>
> This should be forced within the Rolling class in order to avoid any issue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.