You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Edward Capriolo (JIRA)" <ji...@apache.org> on 2010/03/29 18:12:27 UTC
[jira] Created: (HADOOP-6664) fs.inmemory.size.mb not listed in
conf. Cluster setup page gives wrong advice.
fs.inmemory.size.mb not listed in conf. Cluster setup page gives wrong advice.
------------------------------------------------------------------------------
Key: HADOOP-6664
URL: https://issues.apache.org/jira/browse/HADOOP-6664
Project: Hadoop Common
Issue Type: Task
Components: conf, documentation
Affects Versions: 0.20.2
Reporter: Edward Capriolo
http://hadoop.apache.org/common/docs/current/cluster_setup.html
fs.inmemory.size.mb does not appear in any xml file
{noformat}
grep "fs.inmemory.size.mb" ./mapred/mapred-default.xml
[edward@ec src]$ grep "fs.inmemory.size.mb" ./hdfs/hdfs-default.xml
[edward@ec src]$ grep "fs.inmemory.size.mb" ./core/core-default.xml
{noformat}
http://hadoop.apache.org/common/docs/current/cluster_setup.html
Documentation error:
Real-World Cluster Configurations
{noformat}
conf/core-site.xml io.sort.factor 100 More streams merged at once while sorting files.
conf/core-site.xml io.sort.mb 200 Higher memory-limit while sorting data.
{noformat}
core --- io.sort.factor -- should be mapred
core --- io.sort.mb -- should be mapred
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6664) fs.inmemory.size.mb not listed in
conf. Cluster setup page gives wrong advice.
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12851031#action_12851031 ]
Chris Douglas commented on HADOOP-6664:
---------------------------------------
Oh, I see what you mean. {{io.sort.factor}} and {{io.sort.mb}} are also used in SequenceFile, so the config in 0.20 is in core.
> fs.inmemory.size.mb not listed in conf. Cluster setup page gives wrong advice.
> ------------------------------------------------------------------------------
>
> Key: HADOOP-6664
> URL: https://issues.apache.org/jira/browse/HADOOP-6664
> Project: Hadoop Common
> Issue Type: Task
> Components: conf, documentation
> Affects Versions: 0.20.2
> Reporter: Edward Capriolo
>
> http://hadoop.apache.org/common/docs/current/cluster_setup.html
> fs.inmemory.size.mb does not appear in any xml file
> {noformat}
> grep "fs.inmemory.size.mb" ./mapred/mapred-default.xml
> [edward@ec src]$ grep "fs.inmemory.size.mb" ./hdfs/hdfs-default.xml
> [edward@ec src]$ grep "fs.inmemory.size.mb" ./core/core-default.xml
> {noformat}
> http://hadoop.apache.org/common/docs/current/cluster_setup.html
> Documentation error:
> Real-World Cluster Configurations
> {noformat}
> conf/core-site.xml io.sort.factor 100 More streams merged at once while sorting files.
> conf/core-site.xml io.sort.mb 200 Higher memory-limit while sorting data.
> {noformat}
> core --- io.sort.factor -- should be mapred
> core --- io.sort.mb -- should be mapred
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Moved: (HADOOP-6812) fs.inmemory.size.mb not listed in conf.
Cluster setup page gives wrong advice.
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas moved MAPREDUCE-1726 to HADOOP-6812:
--------------------------------------------------
Project: Hadoop Common (was: Hadoop Map/Reduce)
Key: HADOOP-6812 (was: MAPREDUCE-1726)
Affects Version/s: 0.20.2
0.21.0
0.22.0
(was: 0.21.0)
(was: 0.22.0)
(was: 0.20.2)
Component/s: documentation
(was: documentation)
> fs.inmemory.size.mb not listed in conf. Cluster setup page gives wrong advice.
> ------------------------------------------------------------------------------
>
> Key: HADOOP-6812
> URL: https://issues.apache.org/jira/browse/HADOOP-6812
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation
> Affects Versions: 0.20.2, 0.21.0, 0.22.0
> Reporter: Edward Capriolo
> Attachments: M1726-0.patch, M1726-0v20.patch
>
>
> http://hadoop.apache.org/common/docs/current/cluster_setup.html
> fs.inmemory.size.mb does not appear in any xml file
> {noformat}
> grep "fs.inmemory.size.mb" ./mapred/mapred-default.xml
> [edward@ec src]$ grep "fs.inmemory.size.mb" ./hdfs/hdfs-default.xml
> [edward@ec src]$ grep "fs.inmemory.size.mb" ./core/core-default.xml
> {noformat}
> http://hadoop.apache.org/common/docs/current/cluster_setup.html
> Documentation error:
> Real-World Cluster Configurations
> {noformat}
> conf/core-site.xml io.sort.factor 100 More streams merged at once while sorting files.
> conf/core-site.xml io.sort.mb 200 Higher memory-limit while sorting data.
> {noformat}
> core --- io.sort.factor -- should be mapred
> core --- io.sort.mb -- should be mapred
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6812) fs.inmemory.size.mb not listed in
conf. Cluster setup page gives wrong advice.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12928500#action_12928500 ]
Hadoop QA commented on HADOOP-6812:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12442759/M1726-0v20.patch
against trunk revision 1031422.
+1 @author. The patch does not contain any @author tags.
+0 tests included. The patch appears to be a documentation patch that doesn't require tests.
-1 patch. The patch command could not apply the patch.
Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/32//console
This message is automatically generated.
> fs.inmemory.size.mb not listed in conf. Cluster setup page gives wrong advice.
> ------------------------------------------------------------------------------
>
> Key: HADOOP-6812
> URL: https://issues.apache.org/jira/browse/HADOOP-6812
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation
> Affects Versions: 0.20.2, 0.21.0, 0.22.0
> Reporter: Edward Capriolo
> Attachments: M1726-0.patch, M1726-0v20.patch
>
>
> http://hadoop.apache.org/common/docs/current/cluster_setup.html
> fs.inmemory.size.mb does not appear in any xml file
> {noformat}
> grep "fs.inmemory.size.mb" ./mapred/mapred-default.xml
> [edward@ec src]$ grep "fs.inmemory.size.mb" ./hdfs/hdfs-default.xml
> [edward@ec src]$ grep "fs.inmemory.size.mb" ./core/core-default.xml
> {noformat}
> http://hadoop.apache.org/common/docs/current/cluster_setup.html
> Documentation error:
> Real-World Cluster Configurations
> {noformat}
> conf/core-site.xml io.sort.factor 100 More streams merged at once while sorting files.
> conf/core-site.xml io.sort.mb 200 Higher memory-limit while sorting data.
> {noformat}
> core --- io.sort.factor -- should be mapred
> core --- io.sort.mb -- should be mapred
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Reopened: (HADOOP-6664) fs.inmemory.size.mb not listed in
conf. Cluster setup page gives wrong advice.
Posted by "Edward Capriolo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward Capriolo reopened HADOOP-6664:
-------------------------------------
In the future, can you please not close a ticket before I even have a chance to reply.
1) the generated documentation on the site is wrong.
2) the generated xml files in the src directory are putting variables in the wrong files.
People who are not 'In the know' will put configuration variables in the wrong file and not get the effect they desire.
> fs.inmemory.size.mb not listed in conf. Cluster setup page gives wrong advice.
> ------------------------------------------------------------------------------
>
> Key: HADOOP-6664
> URL: https://issues.apache.org/jira/browse/HADOOP-6664
> Project: Hadoop Common
> Issue Type: Task
> Components: conf, documentation
> Affects Versions: 0.20.2
> Reporter: Edward Capriolo
>
> http://hadoop.apache.org/common/docs/current/cluster_setup.html
> fs.inmemory.size.mb does not appear in any xml file
> {noformat}
> grep "fs.inmemory.size.mb" ./mapred/mapred-default.xml
> [edward@ec src]$ grep "fs.inmemory.size.mb" ./hdfs/hdfs-default.xml
> [edward@ec src]$ grep "fs.inmemory.size.mb" ./core/core-default.xml
> {noformat}
> http://hadoop.apache.org/common/docs/current/cluster_setup.html
> Documentation error:
> Real-World Cluster Configurations
> {noformat}
> conf/core-site.xml io.sort.factor 100 More streams merged at once while sorting files.
> conf/core-site.xml io.sort.mb 200 Higher memory-limit while sorting data.
> {noformat}
> core --- io.sort.factor -- should be mapred
> core --- io.sort.mb -- should be mapred
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6812) fs.inmemory.size.mb not listed in
conf. Cluster setup page gives wrong advice.
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876026#action_12876026 ]
Chris Douglas commented on HADOOP-6812:
---------------------------------------
Moved back to Common after MAPREDUCE-1404
> fs.inmemory.size.mb not listed in conf. Cluster setup page gives wrong advice.
> ------------------------------------------------------------------------------
>
> Key: HADOOP-6812
> URL: https://issues.apache.org/jira/browse/HADOOP-6812
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation
> Affects Versions: 0.20.2, 0.21.0, 0.22.0
> Reporter: Edward Capriolo
> Attachments: M1726-0.patch, M1726-0v20.patch
>
>
> http://hadoop.apache.org/common/docs/current/cluster_setup.html
> fs.inmemory.size.mb does not appear in any xml file
> {noformat}
> grep "fs.inmemory.size.mb" ./mapred/mapred-default.xml
> [edward@ec src]$ grep "fs.inmemory.size.mb" ./hdfs/hdfs-default.xml
> [edward@ec src]$ grep "fs.inmemory.size.mb" ./core/core-default.xml
> {noformat}
> http://hadoop.apache.org/common/docs/current/cluster_setup.html
> Documentation error:
> Real-World Cluster Configurations
> {noformat}
> conf/core-site.xml io.sort.factor 100 More streams merged at once while sorting files.
> conf/core-site.xml io.sort.mb 200 Higher memory-limit while sorting data.
> {noformat}
> core --- io.sort.factor -- should be mapred
> core --- io.sort.mb -- should be mapred
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6664) fs.inmemory.size.mb not listed in
conf. Cluster setup page gives wrong advice.
Posted by "Edward Capriolo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859044#action_12859044 ]
Edward Capriolo commented on HADOOP-6664:
-----------------------------------------
If I understandard correctly the docs for current are based on current stable 0.20.2. Current stable does not use fs.inmemory.size.mb.
http://hadoop.apache.org/common/docs/current/cluster_setup.html. Under real world configurations
{noformat}
conf/core-site.xml fs.inmemory.size.mb 200 Larger amount of memory allocated for the in-memory file-system used to merge map-outputs at the reduces.
{noformat}
As to "io.sort.factor and io.sort.mb"
They both appear in mapred-default.xml
{noformat}
[edward@ec src]$ grep -R "io.sort.factor" */*.xml
mapred/mapred-default.xml: <name>io.sort.factor</name>
{noformat}
They should be in core-default.xml (only), or in both core-default.xml and mapred-default.conf.
Think about the end user. An end user might read a blog that states, "io.sort.factor is a magic tune set this to XXXX for awesome performance". Which file should end user put this variable in?
{noformat}
grep -R "io.sort.factor" */*.xml
mapred/mapred-default.xml: <name>io.sort.factor</name>
{noformat}
End user thinks, "Since I found this variable in mapred-default.xml it makese sense that I should override it in mapred-site.xml"
The user puts the variable in the wrong place, because end user has no (easy) way of knowing that SequenceFile uses io.sort.factor or io.sort.mb. Does that make sense?
> fs.inmemory.size.mb not listed in conf. Cluster setup page gives wrong advice.
> ------------------------------------------------------------------------------
>
> Key: HADOOP-6664
> URL: https://issues.apache.org/jira/browse/HADOOP-6664
> Project: Hadoop Common
> Issue Type: Task
> Components: conf, documentation
> Affects Versions: 0.20.2
> Reporter: Edward Capriolo
>
> http://hadoop.apache.org/common/docs/current/cluster_setup.html
> fs.inmemory.size.mb does not appear in any xml file
> {noformat}
> grep "fs.inmemory.size.mb" ./mapred/mapred-default.xml
> [edward@ec src]$ grep "fs.inmemory.size.mb" ./hdfs/hdfs-default.xml
> [edward@ec src]$ grep "fs.inmemory.size.mb" ./core/core-default.xml
> {noformat}
> http://hadoop.apache.org/common/docs/current/cluster_setup.html
> Documentation error:
> Real-World Cluster Configurations
> {noformat}
> conf/core-site.xml io.sort.factor 100 More streams merged at once while sorting files.
> conf/core-site.xml io.sort.mb 200 Higher memory-limit while sorting data.
> {noformat}
> core --- io.sort.factor -- should be mapred
> core --- io.sort.mb -- should be mapred
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HADOOP-6664) fs.inmemory.size.mb not listed in
conf. Cluster setup page gives wrong advice.
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas resolved HADOOP-6664.
-----------------------------------
Resolution: Invalid
> fs.inmemory.size.mb not listed in conf. Cluster setup page gives wrong advice.
> ------------------------------------------------------------------------------
>
> Key: HADOOP-6664
> URL: https://issues.apache.org/jira/browse/HADOOP-6664
> Project: Hadoop Common
> Issue Type: Task
> Components: conf, documentation
> Affects Versions: 0.20.2
> Reporter: Edward Capriolo
>
> http://hadoop.apache.org/common/docs/current/cluster_setup.html
> fs.inmemory.size.mb does not appear in any xml file
> {noformat}
> grep "fs.inmemory.size.mb" ./mapred/mapred-default.xml
> [edward@ec src]$ grep "fs.inmemory.size.mb" ./hdfs/hdfs-default.xml
> [edward@ec src]$ grep "fs.inmemory.size.mb" ./core/core-default.xml
> {noformat}
> http://hadoop.apache.org/common/docs/current/cluster_setup.html
> Documentation error:
> Real-World Cluster Configurations
> {noformat}
> conf/core-site.xml io.sort.factor 100 More streams merged at once while sorting files.
> conf/core-site.xml io.sort.mb 200 Higher memory-limit while sorting data.
> {noformat}
> core --- io.sort.factor -- should be mapred
> core --- io.sort.mb -- should be mapred
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6664) fs.inmemory.size.mb not listed in
conf. Cluster setup page gives wrong advice.
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12851001#action_12851001 ]
Chris Douglas commented on HADOOP-6664:
---------------------------------------
bq. fs.inmemory.size.mb does not appear in any xml file
It isn't used in the source. HADOOP-3446
{quote}
core — io.sort.factor - should be mapred
core — io.sort.mb - should be mapred
{quote}
I don't follow. These are the correct names in 0.20.2, no?
> fs.inmemory.size.mb not listed in conf. Cluster setup page gives wrong advice.
> ------------------------------------------------------------------------------
>
> Key: HADOOP-6664
> URL: https://issues.apache.org/jira/browse/HADOOP-6664
> Project: Hadoop Common
> Issue Type: Task
> Components: conf, documentation
> Affects Versions: 0.20.2
> Reporter: Edward Capriolo
>
> http://hadoop.apache.org/common/docs/current/cluster_setup.html
> fs.inmemory.size.mb does not appear in any xml file
> {noformat}
> grep "fs.inmemory.size.mb" ./mapred/mapred-default.xml
> [edward@ec src]$ grep "fs.inmemory.size.mb" ./hdfs/hdfs-default.xml
> [edward@ec src]$ grep "fs.inmemory.size.mb" ./core/core-default.xml
> {noformat}
> http://hadoop.apache.org/common/docs/current/cluster_setup.html
> Documentation error:
> Real-World Cluster Configurations
> {noformat}
> conf/core-site.xml io.sort.factor 100 More streams merged at once while sorting files.
> conf/core-site.xml io.sort.mb 200 Higher memory-limit while sorting data.
> {noformat}
> core --- io.sort.factor -- should be mapred
> core --- io.sort.mb -- should be mapred
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6664) fs.inmemory.size.mb not listed in
conf. Cluster setup page gives wrong advice.
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859024#action_12859024 ]
Chris Douglas commented on HADOOP-6664:
---------------------------------------
Sorry for closing the issue prematurely, but I'm still unclear on what this issue is about. It sounded like you were saying that {{io.sort.factor}} and {{io.sort.mb}} belong in mapred-default.xml rather than core-default.xml, which I thought I'd answered by noting that these parameters are also used in o.a.h.io.SequenceFile (which is in core, not mapred). Given that {{fs.inmemory.size.mb}} is unused, that it doesn't appear in the default configs is also correct.
bq. the generated documentation on the site is wrong.
bq. the generated xml files in the src directory are putting variables in the wrong files.
How? Can you either explain what is "wrong" or post a patch correcting the error?
> fs.inmemory.size.mb not listed in conf. Cluster setup page gives wrong advice.
> ------------------------------------------------------------------------------
>
> Key: HADOOP-6664
> URL: https://issues.apache.org/jira/browse/HADOOP-6664
> Project: Hadoop Common
> Issue Type: Task
> Components: conf, documentation
> Affects Versions: 0.20.2
> Reporter: Edward Capriolo
>
> http://hadoop.apache.org/common/docs/current/cluster_setup.html
> fs.inmemory.size.mb does not appear in any xml file
> {noformat}
> grep "fs.inmemory.size.mb" ./mapred/mapred-default.xml
> [edward@ec src]$ grep "fs.inmemory.size.mb" ./hdfs/hdfs-default.xml
> [edward@ec src]$ grep "fs.inmemory.size.mb" ./core/core-default.xml
> {noformat}
> http://hadoop.apache.org/common/docs/current/cluster_setup.html
> Documentation error:
> Real-World Cluster Configurations
> {noformat}
> conf/core-site.xml io.sort.factor 100 More streams merged at once while sorting files.
> conf/core-site.xml io.sort.mb 200 Higher memory-limit while sorting data.
> {noformat}
> core --- io.sort.factor -- should be mapred
> core --- io.sort.mb -- should be mapred
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6664) fs.inmemory.size.mb not listed in
conf. Cluster setup page gives wrong advice.
Posted by "Edward Capriolo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12851053#action_12851053 ]
Edward Capriolo commented on HADOOP-6664:
-----------------------------------------
It is confusing to me. Usually I determine which configuration a variable should go into by looking at the default-xml files in <hadoop>/src. io.sort.factor and io.sort.mb are specified in mapred. They should either be in both or just in core, correct?
> fs.inmemory.size.mb not listed in conf. Cluster setup page gives wrong advice.
> ------------------------------------------------------------------------------
>
> Key: HADOOP-6664
> URL: https://issues.apache.org/jira/browse/HADOOP-6664
> Project: Hadoop Common
> Issue Type: Task
> Components: conf, documentation
> Affects Versions: 0.20.2
> Reporter: Edward Capriolo
>
> http://hadoop.apache.org/common/docs/current/cluster_setup.html
> fs.inmemory.size.mb does not appear in any xml file
> {noformat}
> grep "fs.inmemory.size.mb" ./mapred/mapred-default.xml
> [edward@ec src]$ grep "fs.inmemory.size.mb" ./hdfs/hdfs-default.xml
> [edward@ec src]$ grep "fs.inmemory.size.mb" ./core/core-default.xml
> {noformat}
> http://hadoop.apache.org/common/docs/current/cluster_setup.html
> Documentation error:
> Real-World Cluster Configurations
> {noformat}
> conf/core-site.xml io.sort.factor 100 More streams merged at once while sorting files.
> conf/core-site.xml io.sort.mb 200 Higher memory-limit while sorting data.
> {noformat}
> core --- io.sort.factor -- should be mapred
> core --- io.sort.mb -- should be mapred
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.