You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Arun C Murthy (JIRA)" <ji...@apache.org> on 2007/07/06 18:27:04 UTC
[jira] Created: (HADOOP-1570) Add a per-job configuration knob to
control loading of native hadoop libraries
Add a per-job configuration knob to control loading of native hadoop libraries
-------------------------------------------------------------------------------
Key: HADOOP-1570
URL: https://issues.apache.org/jira/browse/HADOOP-1570
Project: Hadoop
Issue Type: New Feature
Components: io
Affects Versions: 0.13.0, 0.12.3
Reporter: Arun C Murthy
Assignee: Arun C Murthy
Fix For: 0.14.0
As it exists today, native-hadoop libraries are loaded automatically if libhadoop.so is present; however we have sporadically seen issues (HADOOP-1545) since native direct-buffers aren't very well understood. The only way to switch off usage of these is to remove the native libraries which is a maintenence issue for large clusters...
Hence I propose we add a per-job config knob: {{hadoop.native.lib}} (set to {{true}} by default) which can be used to control usage of native libraries even when the libraries are present e.g. we can have hadoop installed with native libraries present and then use this knob to switch off their usage in rare cases we see issues with them; thus aiding maintenence.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1570) Add a per-job configuration knob to
control loading of native hadoop libraries
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated HADOOP-1570:
----------------------------------
Status: Patch Available (was: Open)
> Add a per-job configuration knob to control loading of native hadoop libraries
> -------------------------------------------------------------------------------
>
> Key: HADOOP-1570
> URL: https://issues.apache.org/jira/browse/HADOOP-1570
> Project: Hadoop
> Issue Type: New Feature
> Components: io
> Affects Versions: 0.13.0, 0.12.3
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Fix For: 0.14.0
>
> Attachments: HADOOP-1570_1_20070709.patch, HADOOP-1570_2_20070713.patch
>
>
> As it exists today, native-hadoop libraries are loaded automatically if libhadoop.so is present; however we have sporadically seen issues (HADOOP-1545) since native direct-buffers aren't very well understood. The only way to switch off usage of these is to remove the native libraries which is a maintenence issue for large clusters...
> Hence I propose we add a per-job config knob: {{hadoop.native.lib}} (set to {{true}} by default) which can be used to control usage of native libraries even when the libraries are present e.g. we can have hadoop installed with native libraries present and then use this knob to switch off their usage in rare cases we see issues with them; thus aiding maintenence.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1570) Add a per-job configuration knob to
control loading of native hadoop libraries
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated HADOOP-1570:
----------------------------------
Attachment: HADOOP-1570_2_20070713.patch
Incorporated Doug's comments, thanks for the f/b!
> Add a per-job configuration knob to control loading of native hadoop libraries
> -------------------------------------------------------------------------------
>
> Key: HADOOP-1570
> URL: https://issues.apache.org/jira/browse/HADOOP-1570
> Project: Hadoop
> Issue Type: New Feature
> Components: io
> Affects Versions: 0.12.3, 0.13.0
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Fix For: 0.14.0
>
> Attachments: HADOOP-1570_1_20070709.patch, HADOOP-1570_2_20070713.patch
>
>
> As it exists today, native-hadoop libraries are loaded automatically if libhadoop.so is present; however we have sporadically seen issues (HADOOP-1545) since native direct-buffers aren't very well understood. The only way to switch off usage of these is to remove the native libraries which is a maintenence issue for large clusters...
> Hence I propose we add a per-job config knob: {{hadoop.native.lib}} (set to {{true}} by default) which can be used to control usage of native libraries even when the libraries are present e.g. we can have hadoop installed with native libraries present and then use this knob to switch off their usage in rare cases we see issues with them; thus aiding maintenence.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1570) Add a per-job configuration knob to
control loading of native hadoop libraries
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated HADOOP-1570:
----------------------------------
Attachment: HADOOP-1570_1_20070709.patch
Straight-forward patch...
> Add a per-job configuration knob to control loading of native hadoop libraries
> -------------------------------------------------------------------------------
>
> Key: HADOOP-1570
> URL: https://issues.apache.org/jira/browse/HADOOP-1570
> Project: Hadoop
> Issue Type: New Feature
> Components: io
> Affects Versions: 0.12.3, 0.13.0
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Fix For: 0.14.0
>
> Attachments: HADOOP-1570_1_20070709.patch
>
>
> As it exists today, native-hadoop libraries are loaded automatically if libhadoop.so is present; however we have sporadically seen issues (HADOOP-1545) since native direct-buffers aren't very well understood. The only way to switch off usage of these is to remove the native libraries which is a maintenence issue for large clusters...
> Hence I propose we add a per-job config knob: {{hadoop.native.lib}} (set to {{true}} by default) which can be used to control usage of native libraries even when the libraries are present e.g. we can have hadoop installed with native libraries present and then use this knob to switch off their usage in rare cases we see issues with them; thus aiding maintenence.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1570) Add a per-job configuration knob to
control loading of native hadoop libraries
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated HADOOP-1570:
---------------------------------
Status: Open (was: Patch Available)
This looks great, except the methods to set the configuration property don't belong on JobConf. JobConf should only have methods specific to jobs, settings read by the MapReduce kernel. We've not been great about enforcing this in the past, but don't want JobConf to just turn into a heap of settings. Rather these should be static methods on related code, like util.NativeCodeLoader.
> Add a per-job configuration knob to control loading of native hadoop libraries
> -------------------------------------------------------------------------------
>
> Key: HADOOP-1570
> URL: https://issues.apache.org/jira/browse/HADOOP-1570
> Project: Hadoop
> Issue Type: New Feature
> Components: io
> Affects Versions: 0.13.0, 0.12.3
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Fix For: 0.14.0
>
> Attachments: HADOOP-1570_1_20070709.patch
>
>
> As it exists today, native-hadoop libraries are loaded automatically if libhadoop.so is present; however we have sporadically seen issues (HADOOP-1545) since native direct-buffers aren't very well understood. The only way to switch off usage of these is to remove the native libraries which is a maintenence issue for large clusters...
> Hence I propose we add a per-job config knob: {{hadoop.native.lib}} (set to {{true}} by default) which can be used to control usage of native libraries even when the libraries are present e.g. we can have hadoop installed with native libraries present and then use this knob to switch off their usage in rare cases we see issues with them; thus aiding maintenence.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1570) Add a per-job configuration knob to
control loading of native hadoop libraries
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12512444 ]
Hudson commented on HADOOP-1570:
--------------------------------
Integrated in Hadoop-Nightly #154 (See [http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/154/])
> Add a per-job configuration knob to control loading of native hadoop libraries
> -------------------------------------------------------------------------------
>
> Key: HADOOP-1570
> URL: https://issues.apache.org/jira/browse/HADOOP-1570
> Project: Hadoop
> Issue Type: New Feature
> Components: io
> Affects Versions: 0.12.3, 0.13.0
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Fix For: 0.14.0
>
> Attachments: HADOOP-1570_1_20070709.patch, HADOOP-1570_2_20070713.patch
>
>
> As it exists today, native-hadoop libraries are loaded automatically if libhadoop.so is present; however we have sporadically seen issues (HADOOP-1545) since native direct-buffers aren't very well understood. The only way to switch off usage of these is to remove the native libraries which is a maintenence issue for large clusters...
> Hence I propose we add a per-job config knob: {{hadoop.native.lib}} (set to {{true}} by default) which can be used to control usage of native libraries even when the libraries are present e.g. we can have hadoop installed with native libraries present and then use this knob to switch off their usage in rare cases we see issues with them; thus aiding maintenence.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1570) Add a per-job configuration knob to
control loading of native hadoop libraries
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12511069 ]
Hadoop QA commented on HADOOP-1570:
-----------------------------------
+1
http://issues.apache.org/jira/secure/attachment/12361402/HADOOP-1570_1_20070709.patch applied and successfully tested against trunk revision r554144.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/374/testReport/
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/374/console
> Add a per-job configuration knob to control loading of native hadoop libraries
> -------------------------------------------------------------------------------
>
> Key: HADOOP-1570
> URL: https://issues.apache.org/jira/browse/HADOOP-1570
> Project: Hadoop
> Issue Type: New Feature
> Components: io
> Affects Versions: 0.12.3, 0.13.0
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Fix For: 0.14.0
>
> Attachments: HADOOP-1570_1_20070709.patch
>
>
> As it exists today, native-hadoop libraries are loaded automatically if libhadoop.so is present; however we have sporadically seen issues (HADOOP-1545) since native direct-buffers aren't very well understood. The only way to switch off usage of these is to remove the native libraries which is a maintenence issue for large clusters...
> Hence I propose we add a per-job config knob: {{hadoop.native.lib}} (set to {{true}} by default) which can be used to control usage of native libraries even when the libraries are present e.g. we can have hadoop installed with native libraries present and then use this knob to switch off their usage in rare cases we see issues with them; thus aiding maintenence.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1570) Add a per-job configuration knob to
control loading of native hadoop libraries
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated HADOOP-1570:
---------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this. Thanks, Arun!
> Add a per-job configuration knob to control loading of native hadoop libraries
> -------------------------------------------------------------------------------
>
> Key: HADOOP-1570
> URL: https://issues.apache.org/jira/browse/HADOOP-1570
> Project: Hadoop
> Issue Type: New Feature
> Components: io
> Affects Versions: 0.12.3, 0.13.0
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Fix For: 0.14.0
>
> Attachments: HADOOP-1570_1_20070709.patch, HADOOP-1570_2_20070713.patch
>
>
> As it exists today, native-hadoop libraries are loaded automatically if libhadoop.so is present; however we have sporadically seen issues (HADOOP-1545) since native direct-buffers aren't very well understood. The only way to switch off usage of these is to remove the native libraries which is a maintenence issue for large clusters...
> Hence I propose we add a per-job config knob: {{hadoop.native.lib}} (set to {{true}} by default) which can be used to control usage of native libraries even when the libraries are present e.g. we can have hadoop installed with native libraries present and then use this knob to switch off their usage in rare cases we see issues with them; thus aiding maintenence.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1570) Add a per-job configuration knob to
control loading of native hadoop libraries
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated HADOOP-1570:
----------------------------------
Status: Patch Available (was: Open)
> Add a per-job configuration knob to control loading of native hadoop libraries
> -------------------------------------------------------------------------------
>
> Key: HADOOP-1570
> URL: https://issues.apache.org/jira/browse/HADOOP-1570
> Project: Hadoop
> Issue Type: New Feature
> Components: io
> Affects Versions: 0.13.0, 0.12.3
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Fix For: 0.14.0
>
> Attachments: HADOOP-1570_1_20070709.patch
>
>
> As it exists today, native-hadoop libraries are loaded automatically if libhadoop.so is present; however we have sporadically seen issues (HADOOP-1545) since native direct-buffers aren't very well understood. The only way to switch off usage of these is to remove the native libraries which is a maintenence issue for large clusters...
> Hence I propose we add a per-job config knob: {{hadoop.native.lib}} (set to {{true}} by default) which can be used to control usage of native libraries even when the libraries are present e.g. we can have hadoop installed with native libraries present and then use this knob to switch off their usage in rare cases we see issues with them; thus aiding maintenence.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.