You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Andrew Purtell (Created) (JIRA)" <ji...@apache.org> on 2012/01/11 02:12:39 UTC
[jira] [Created] (MAPREDUCE-3654) MiniMRYarnCluster should set
MASTER_ADDRESS to "local"
MiniMRYarnCluster should set MASTER_ADDRESS to "local"
------------------------------------------------------
Key: MAPREDUCE-3654
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3654
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 0.23.1
Reporter: Andrew Purtell
I needed to make the attached change in order for MiniMRCluster based HBase tests to get past job client initialization.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3654) MiniMRYarnCluster should set
MASTER_ADDRESS to "local"
Posted by "Andrew Purtell (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13189575#comment-13189575 ]
Andrew Purtell commented on MAPREDUCE-3654:
-------------------------------------------
I can accept the resolution for this issue in particular. However, with MRv1 we could simply instantiate a MiniMRCluster and it would start without additional configuration. It is the Hadoop code that delegates MiniMRCluster to the MiniYarnMRCluster. Our client code shouldn't have to set any arguably internal configuration variables to avoid cryptic MiniMRCluster failures.
Mahadev helped us update our POM. I applied these changes locally and still all MiniMRCluster based HBase tests fail to launch. Have you been able to successfully run the HBase unit tests like TestTableMapReduce successfully with 0.23.1-SNAPSHOT?
> MiniMRYarnCluster should set MASTER_ADDRESS to "local"
> ------------------------------------------------------
>
> Key: MAPREDUCE-3654
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3654
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, test
> Affects Versions: 0.23.1
> Reporter: Andrew Purtell
> Assignee: Andrew Purtell
> Priority: Critical
> Attachments: MAPREDUCE-3654.patch
>
>
> I needed to make the attached change in order for MiniMRCluster based HBase tests to get past job client initialization.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3654) MiniMRYarnCluster should set
MASTER_ADDRESS to "local"
Posted by "Hitesh Shah (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hitesh Shah resolved MAPREDUCE-3654.
------------------------------------
Resolution: Invalid
Looking more into the hbase test, a couple of issues were found ( pom fix to depend on the required jar so as to find the YarnClientProtocol Provider and a change required to the job client submission to use the conf from the initialized MiniMRYarnCluster object ).
Marking this as invalid as it seems to me that the fix needs to be done in the hbase test code.
> MiniMRYarnCluster should set MASTER_ADDRESS to "local"
> ------------------------------------------------------
>
> Key: MAPREDUCE-3654
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3654
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, test
> Affects Versions: 0.23.1
> Reporter: Andrew Purtell
> Assignee: Andrew Purtell
> Priority: Critical
> Attachments: MAPREDUCE-3654.patch
>
>
> I needed to make the attached change in order for MiniMRCluster based HBase tests to get past job client initialization.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3654) MiniMRYarnCluster should set
MASTER_ADDRESS to "local"
Posted by "Andrew Purtell (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated MAPREDUCE-3654:
--------------------------------------
Attachment: MAPREDUCE-3654.patch
> MiniMRYarnCluster should set MASTER_ADDRESS to "local"
> ------------------------------------------------------
>
> Key: MAPREDUCE-3654
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3654
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
> Affects Versions: 0.23.1
> Reporter: Andrew Purtell
> Attachments: MAPREDUCE-3654.patch
>
>
> I needed to make the attached change in order for MiniMRCluster based HBase tests to get past job client initialization.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3654) MiniMRYarnCluster should set
MASTER_ADDRESS to "local"
Posted by "Mahadev konar (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-3654:
-------------------------------------
Component/s: mrv2
Priority: Critical (was: Major)
Assignee: Andrew Purtell
> MiniMRYarnCluster should set MASTER_ADDRESS to "local"
> ------------------------------------------------------
>
> Key: MAPREDUCE-3654
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3654
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, test
> Affects Versions: 0.23.1
> Reporter: Andrew Purtell
> Assignee: Andrew Purtell
> Priority: Critical
> Attachments: MAPREDUCE-3654.patch
>
>
> I needed to make the attached change in order for MiniMRCluster based HBase tests to get past job client initialization.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3654) MiniMRYarnCluster should set
MASTER_ADDRESS to "local"
Posted by "Hitesh Shah (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13183826#comment-13183826 ]
Hitesh Shah commented on MAPREDUCE-3654:
----------------------------------------
Nevermind - saw the mail on the user list.
Adding the info here for reference:
2012-01-10 18:01:02,489 INFO [main] mapreduce.TestTableMapReduce(153): Started mrtest
2012-01-10 18:01:02,493 DEBUG [main] mapreduce.Cluster(93): Trying ClientProtocolProvider : org.apache.hadoop.mapred.LocalClientProtocolProvider
2012-01-10 18:01:02,494 INFO [main] mapreduce.Cluster(116): Failed to use org.apache.hadoop.mapred.LocalClientProtocolProvider due to error: Invalid "mapreduce.jobtracker.address" configuration value for LocalJobRunner : "test"
2012-01-10 18:01:02,494 ERROR [main] security.UserGroupInformation(1160): PriviledgedActionException as:apurtell (auth:SIMPLE) cause:java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
2012-01-10 18:01:02,497 INFO [main] hbase.HBaseTestingUtility(1173): Stopping mini mapreduce cluster...
2
> MiniMRYarnCluster should set MASTER_ADDRESS to "local"
> ------------------------------------------------------
>
> Key: MAPREDUCE-3654
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3654
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, test
> Affects Versions: 0.23.1
> Reporter: Andrew Purtell
> Assignee: Andrew Purtell
> Priority: Critical
> Attachments: MAPREDUCE-3654.patch
>
>
> I needed to make the attached change in order for MiniMRCluster based HBase tests to get past job client initialization.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3654) MiniMRYarnCluster should set
MASTER_ADDRESS to "local"
Posted by "Hitesh Shah (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13183831#comment-13183831 ]
Hitesh Shah commented on MAPREDUCE-3654:
----------------------------------------
The service loader for the ClientProtocolProvider should cycle through a list containing both the local and yarn client providers. The debug logs seem to indicate that it only sees the local client provider as an option and fails as the framework is set to yarn.
The fix probably might be in the unit test itself - could you ensure the job/jobconf is being initialized using MiniMrYarnCluster#getConfig ?
> MiniMRYarnCluster should set MASTER_ADDRESS to "local"
> ------------------------------------------------------
>
> Key: MAPREDUCE-3654
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3654
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, test
> Affects Versions: 0.23.1
> Reporter: Andrew Purtell
> Assignee: Andrew Purtell
> Priority: Critical
> Attachments: MAPREDUCE-3654.patch
>
>
> I needed to make the attached change in order for MiniMRCluster based HBase tests to get past job client initialization.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3654) MiniMRYarnCluster should set
MASTER_ADDRESS to "local"
Posted by "Hitesh Shah (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13183832#comment-13183832 ]
Hitesh Shah commented on MAPREDUCE-3654:
----------------------------------------
Forgot to add, there still might be an issue with the service loading as it does not seem to find the yarn client provider.
> MiniMRYarnCluster should set MASTER_ADDRESS to "local"
> ------------------------------------------------------
>
> Key: MAPREDUCE-3654
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3654
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, test
> Affects Versions: 0.23.1
> Reporter: Andrew Purtell
> Assignee: Andrew Purtell
> Priority: Critical
> Attachments: MAPREDUCE-3654.patch
>
>
> I needed to make the attached change in order for MiniMRCluster based HBase tests to get past job client initialization.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3654) MiniMRYarnCluster should set
MASTER_ADDRESS to "local"
Posted by "Hitesh Shah (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13189606#comment-13189606 ]
Hitesh Shah commented on MAPREDUCE-3654:
----------------------------------------
In the TestTableMapReduce test, I ended up doing something along the lines of:
table.getConfiguration().set("mapreduce.framework.name", "yarn");
job = new Job(table.getConfiguration(), "process column contents");
to get around the clientprovider issue which seemed to have the test run successfully but I could not completely verify as there are no logs generated even though the hbase/hadoop log levels are set to debug. However, the above is in any case not the correct solution.
When submitting a job to a cluster, the assumption would be to use the relevant client-side aspects of the cluster's configuration. However in the hbase test, what is happening is that the job and htable objects are being initialized via HbaseTestingUtility.getConfiguration(). The test framework encapsulates the dfs/mr clusters but currently it does set internal configs like the jobtracker address for the test to work. It would need to change to probably use the new cluster factory created as part of MAPREDUCE-3169 to switch between mr1/mr2 and also likely set some config values if the job configs are not initialized using the mrCluster's configuration ( such as jobtracker address for MR1 and for MR2, yarn's resource/node mgr addresses in addition to the framework name being set to yarn).
> MiniMRYarnCluster should set MASTER_ADDRESS to "local"
> ------------------------------------------------------
>
> Key: MAPREDUCE-3654
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3654
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, test
> Affects Versions: 0.23.1
> Reporter: Andrew Purtell
> Assignee: Andrew Purtell
> Priority: Critical
> Attachments: MAPREDUCE-3654.patch
>
>
> I needed to make the attached change in order for MiniMRCluster based HBase tests to get past job client initialization.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3654) MiniMRYarnCluster should set
MASTER_ADDRESS to "local"
Posted by "Mahadev konar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191595#comment-13191595 ]
Mahadev konar commented on MAPREDUCE-3654:
------------------------------------------
@Andrew,
I was able to run the test all fine after applying the patch on HBASE-5212. The only issue being HBASE-5191. Ill have some bandwidth this week to work on 0.23 and HBase issues. Ill update the hbase jiras.
> MiniMRYarnCluster should set MASTER_ADDRESS to "local"
> ------------------------------------------------------
>
> Key: MAPREDUCE-3654
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3654
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, test
> Affects Versions: 0.23.1
> Reporter: Andrew Purtell
> Assignee: Andrew Purtell
> Priority: Critical
> Attachments: MAPREDUCE-3654.patch
>
>
> I needed to make the attached change in order for MiniMRCluster based HBase tests to get past job client initialization.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3654) MiniMRYarnCluster should set
MASTER_ADDRESS to "local"
Posted by "Hitesh Shah (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13183823#comment-13183823 ]
Hitesh Shah commented on MAPREDUCE-3654:
----------------------------------------
The change seems a bit odd as master address which is the jobtracker address should not be referenced in a yarn cluster.
Could you attach logs or add more information as to how this solves the hbase job client init failure?
> MiniMRYarnCluster should set MASTER_ADDRESS to "local"
> ------------------------------------------------------
>
> Key: MAPREDUCE-3654
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3654
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, test
> Affects Versions: 0.23.1
> Reporter: Andrew Purtell
> Assignee: Andrew Purtell
> Priority: Critical
> Attachments: MAPREDUCE-3654.patch
>
>
> I needed to make the attached change in order for MiniMRCluster based HBase tests to get past job client initialization.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira